Multi-purpose Testing Tool Kit

I would like to reveal my secret toolkit which I use to tackle any test challenge.

These three thinking tools are shaped to work on every Testing task from small to big. Sometimes, using these tools makes the difference between the ordinary and the expert testers’ way of thinking.

What is in my kit?

I keep the following three thought tools at hand:

·         Definition of Quality

·         Risk Formula

·         Awareness to ROI

Definition of Quality

Every action that we do as testers should have connection to the quality of the product. We

learn the product, test it and provide information, in order to improve the quality. Otherwise - what is the point?

To decide whether an activity is connected to the quality of the product - we must first align ourselves on “what is quality”.

My preferred definition for quality is “value to someone” (Jerry Weinberg). Cem Kaner adds the extension “who matters”, which is useful for focusing on the needs of the person that we are testing for. The benefit of such a basic definition is that it can be used to criticize less basic baselines of quality, like “adhering to the official requirements” or quantitative measurements which are used in the organization process.

The Risk Calculation Formula

Our testing is an activity to mitigate risk. We test in order to provide information about the quality of the product, so that defects with the higher risk to the product quality will be fixed.  

A basic formula of calculating a risk is:

Risk = Probability multiple by impact

Using this formula helps us to compare between the different risks that we want to address. We will try to focus our work first on areas which have the highest probability of having issues that will have the highest impact on quality.

[Late note: My coulege, Amit Wertheimer point out in the hebrew testing forum of Tapuz  that it is important to remember that the caculation is based on our limited analisys  and not on accurate certain factors ]

Awareness of ROI

Testing is an economic activity. Testing consumes resources and is done in order to supply the one who send us to test with information that he values. Whether it is issues that he chooses to fix or risk assessment that will allow him to determine the shipping time of the product.

ROI - return on investment, means that something we do is “worth it” and does not cost us more than what we will get from it. Of course, since Testing is an activity of learning and gathering information, we can’t know ahead of time whether we will learn something that is “worth it”. Here, performing Risk analysis will help us to decide if a testing activity has a good probability of resulting in positive ROI or not.

Examples

Let’s look at some examples for using the tools:

A tester gets a new version of the SW and has to decide what to test first. The risk formula will guide him by viewing the risk factors: Looking at the Probability of possible failures, he will determine the areas which have higher probability of having defects - areas that been changed. Areas that tend to be buggy and so on... On the other hand he will look at the impact of possible issues that he can test for: what flows are the most critical for the product users, what type of defects are more costly to the firm. The first areas that he will pick to test will be the ones that will calculate the highest risk - probability to fail, multiplied by the impact the quality. These areas will have the most ROI as he assesses them as the most valuable.

This example can be true also when planning a test strategy for a whole product for a whole team.

Discussing Whether to fix a Defect or not

If you consider quality, risk and ROI, you understand that fixing a bug is not always the right choice.

A smart tester will use the risk formula and will guide the discussion to consider the impact on the quality, the probability that a user will be effected by the defect on the one hand, and on the other hand - the probability that fixing the defect will cause other defects. This way we can estimate the total cost of the fix in terms of risk and decide whether the defect fix has positive ROI.

Sometimes, a proposal to fix a defect is actually a choice between different quality attributes. For example, making the product more secure may reduce the performance. Examining the impact on the “someone who matters”, usually our user, will help us make a choice.

ROI is a key thinking tool to examine investments in infrastructure, tools and automation. A realistic estimation of how much the tool will cost - in terms of development, integration, maintenance and fees versus the benefits we expect to gain, such as time saving, additional test coverage, will help us decide whether to “go for it” or not.

Some Critique of the Above

Essentially, all models are wrong, but some are useful - said George E. P. Box. This is true for the thought models that I just described. Since Testing is a learning activity, and learning is unpredictable, sometimes wandering around may be the right thing to do. Low risk and low ROI activity that we barely connect to the product quality in the first place, may lead us to critical information that has high ROI and high risk which is connected to the product quality.

What is your secret tool? Do you use a different version of similar tools? Let me know.

Professional meeting? JeST DO IT

This post is about my experience of organizing Tester’s meeting. It is true  as well for any other meeting, be it Software Developers, Social workers or a meeting of Puppeteers.

The 3rd meeting of JeST – the Jerusalem workshop on SW Testing is just over. Such a meeting is a great opportunity to meet fellow testers, and share experiences and ideas.

I am the proud founder of JeST, together with Shmuel Gershon, Asher Samuels and Alon Waisbard (look in Twitter for @sgershon, @absamuels  and @awaisbard ). I am telling you that because less than a year ago, organizing such a meeting, looked to me as a distant vision, something that will happen in the far future when I will develop the complex skills necessary to organize such an event.

But it doesn’t.

A moment during JeST2

Organizing a meeting is so easy, that I must share how easy it is, with you. The main issue to overcome is the mental block and just do it. To help you overcome this block, I prepared a list of the things that you don’t need, in order to hold such a meeting:

1)      A budget

You will be able to find a place to host a short meeting. You don’t need to rent a conference hall. It can be the room in the company of one of the meeting participants, or one’s home living room. Refreshments can be brought by the host or by the participants.  If it works for you and the potential participants, you can schedule a meeting in a quiet restaurant and address the need for both the location and refreshments.

2)      Sponsors

Since you don’t need a heavy budget, you don’t need an organization to support your meeting. Although I welcome cooperation and networking, which naturally happens in such meetings, I prefer to remain independent and stay away from organizations that try to use the meeting for their own promotion.

A thank you note should be enough as a return of the hosts good will.

3)      Heavy logistics

You don’t need a full-time administrator to organize such a meeting. Split the tasks between a few friends and it will not consume too much time. Use social networks to spread the word. Meetup service can simplify the process even more, in return for a payment of a few bucks.

(Late note: there are many Meetup groups which are allready dedicated to promote events like yours. See for example: ILTechTalks group. All you need is to suggest your event in such group, free of charge)

4)      Professional presenters and presentations

Not everyone who has interesting things to say is a professional presenter and the length of a

JeST3 agenda

presentation does not make it better (most times it’s the opposite). For our meetings, we chose a format of lightning talks followed by a discussion.  One time, we had a “regular” presentation, which was great too, but the most significant part for most of the participants was the discussions that we held.

5)      A large number of participants

Rating is overrated. While it is nice to see many people come to “your” meeting, a large audience has its drawbacks too – the atmosphere is less intimate, it’s harder to facilitate the discussion, and not every participant feels free to express his opinion. When I started to organize the first JeST meeting, I was worried that the meeting would not  draw a “respectful” number of participants.  I did the “Worst case scenario” drill and came to conclusion that if only me and the other 3 folks that shared the idea of organizing the meeting would come, it would not be a waste of time.  At the end the meeting turned out to be a success, also by the number of participants.

All the above are not a “must have” in order to organize your testers meeting. All you need is Testers and a love of testing.