1st column of a series I am writing in "Testing World" magazine(in Hebrew) about thinking patterns that I found useful in our work as Testers.
First column subject is about the definition of quality.

Link to the full magazine issue: https://bit.ly/39Wm3BV

Experience report of a Tester in (fr)agile world

The following video shows an experience report of a tester working in different scrum team by my friend Yaakov Morgenstein, a Tester and scrum master. 

This short talk given during the latest Jerusalem SW testing meetup that  Shmuel Gershon and I organize  from time to time.

A nice taste from a lightning talks meetup.

Enjoy!

White elephants

 According to the legend, the kings of Siam would gift a white elephant to courtiers who had rendered themselves obnoxious, in order to ruin the recipient by the cost of its maintenance. Based on that, the term “white elephant” is used to describe a possession which its owner cannot dispose of and whose cost, particularly that of maintenance, is out of proportion to its usefulness.

It’s very easy to spot a real white elephant. They are rare and stand out from the background. However, Testing projects tend to attract lots of white elephants which are less easy to identify.

I would like to share with you my observation of the white elephants life cycle in testing projects.

Birth

The birth of white elephants is usually a result of a match between cool ideas and resources. A future parent with a cool idea pitches it to the herd, which are carried away by it. The cooler the idea is, the bigger the resources allocated to create and maintain it. Many times the declared goal of the cub creation is to save efforts in the long term. Different ideas takes different forms. Common examples are tools that automate manual tasks, enforce a better process, save testers from writing code, etc.

Childhood

The early days of the white elephant are exciting. Brainstorming happens, designs are created and an

artist is invited.  He is freed from his usual less cool tasks, to give life and nurture the young elephant. New technologies and approaches are learnt. A lot of fun happens all around.

On the elephant’s first birthday, the POC is ready and a demo is presented to the herd. The herd is

excited. The elephant shows a great potential. The proud parents beam and relatives from distant places come to observe the miracle taking his first steps.

Youth

The elephant’s caretakers keep raising the cub. They take him outside so he can start living independently and begin contributing to the community. Often, the community members find out that the young beast is not stable enough. The community is very happy to receive him and show a high tolerance of his childish behavior in the early stages. This tolerance may be challenged by the elephant’s habit of requiring too much attention and performing incomplete work.

Adulthood

While the white elephant allegedly reached adulthood it still does not act like an adult.  On the one hand, the people enjoy his company and the cool atmosphere that he spreads around him and they are still hoping he fulfills his potential. On the other hand, each time they try to use him, they are disappointed. Often they continue using the ordinary work elephants and return the white one to his caretaker who still spends too much time nurturing him. On the outside, the picture looks much more ideal. Every guest is taken to see a demo of the white elephant working. His caretaker takes him to conferences within and outside the organization. Prestige is acquired as well as prizes. People still hope the elephant will fulfill the purpose of its creation, despite the growing concerns.

Death 

Most of the white elephants dies naturally when the community gets rid of them. The community reaches a point where they lose hope of benefiting from the contribution they had imagined and understand that they do not gain real benefits from the white elephant. Getting rid of the beast is very painful to the people, but they understand that there is no choice. Many times, in order to ease the pain, the birth of elephant 2.0 is declared. The parents plan to make him a better elephant that won’t turn out to be a white one. A soon as the white elephant is gotten rid of, the pain is reduced. 

In very rare cases, instead of killing the white elephant, the people are able to transform it. In most cases such attempts only consume more resources and delay the inevitable.

Spotting White elephants

No modern white elephant is designed to be one. The guy who can help us spot a white elephant, so we can kill him in early stages, is called Roi, or in his full long boring name: Return On Investment calculation.

When the community forgets to invite Roi to the party, no one is there to set real expectations of the elephant. The most romantic time, the birth of the cool idea, is also the time to make the non-emotional calculations of ROI.  It is the time to think about the possible lowlights the new approach is based on.

I suggest that as much as our idea is cool and looks good, so should our ROI calculation be ruthless, to make up for our strong bias in support of our idea.

Doing the ROI calculation once and forgeting about it, is a common mistake. The unexpected happens and turns our elephant into a white one. As more resources are consumed working on the idea, we are more biased towards it and we start to “throw good money after bad”. We tend to take into account the sunk costs. which is not a best decision making practice.

Monitoring the ROI, and continuously comparing our expectation to the reality, can help us identify our white elephant early, so we will be able to get rid of him before he consumes too much of our resources.

Can you spot the white elephant near you?

Multi-purpose Testing Tool Kit

I would like to reveal my secret toolkit which I use to tackle any test challenge.

These three thinking tools are shaped to work on every Testing task from small to big. Sometimes, using these tools makes the difference between the ordinary and the expert testers’ way of thinking.

What is in my kit?

I keep the following three thought tools at hand:

·         Definition of Quality

·         Risk Formula

·         Awareness to ROI

Definition of Quality

Every action that we do as testers should have connection to the quality of the product. We

learn the product, test it and provide information, in order to improve the quality. Otherwise - what is the point?

To decide whether an activity is connected to the quality of the product - we must first align ourselves on “what is quality”.

My preferred definition for quality is “value to someone” (Jerry Weinberg). Cem Kaner adds the extension “who matters”, which is useful for focusing on the needs of the person that we are testing for. The benefit of such a basic definition is that it can be used to criticize less basic baselines of quality, like “adhering to the official requirements” or quantitative measurements which are used in the organization process.

The Risk Calculation Formula

Our testing is an activity to mitigate risk. We test in order to provide information about the quality of the product, so that defects with the higher risk to the product quality will be fixed.  

A basic formula of calculating a risk is:

Risk = Probability multiple by impact

Using this formula helps us to compare between the different risks that we want to address. We will try to focus our work first on areas which have the highest probability of having issues that will have the highest impact on quality.

[Late note: My coulege, Amit Wertheimer point out in the hebrew testing forum of Tapuz  that it is important to remember that the caculation is based on our limited analisys  and not on accurate certain factors ]

Awareness of ROI

Testing is an economic activity. Testing consumes resources and is done in order to supply the one who send us to test with information that he values. Whether it is issues that he chooses to fix or risk assessment that will allow him to determine the shipping time of the product.

ROI - return on investment, means that something we do is “worth it” and does not cost us more than what we will get from it. Of course, since Testing is an activity of learning and gathering information, we can’t know ahead of time whether we will learn something that is “worth it”. Here, performing Risk analysis will help us to decide if a testing activity has a good probability of resulting in positive ROI or not.

Examples

Let’s look at some examples for using the tools:

A tester gets a new version of the SW and has to decide what to test first. The risk formula will guide him by viewing the risk factors: Looking at the Probability of possible failures, he will determine the areas which have higher probability of having defects - areas that been changed. Areas that tend to be buggy and so on... On the other hand he will look at the impact of possible issues that he can test for: what flows are the most critical for the product users, what type of defects are more costly to the firm. The first areas that he will pick to test will be the ones that will calculate the highest risk - probability to fail, multiplied by the impact the quality. These areas will have the most ROI as he assesses them as the most valuable.

This example can be true also when planning a test strategy for a whole product for a whole team.

Discussing Whether to fix a Defect or not

If you consider quality, risk and ROI, you understand that fixing a bug is not always the right choice.

A smart tester will use the risk formula and will guide the discussion to consider the impact on the quality, the probability that a user will be effected by the defect on the one hand, and on the other hand - the probability that fixing the defect will cause other defects. This way we can estimate the total cost of the fix in terms of risk and decide whether the defect fix has positive ROI.

Sometimes, a proposal to fix a defect is actually a choice between different quality attributes. For example, making the product more secure may reduce the performance. Examining the impact on the “someone who matters”, usually our user, will help us make a choice.

ROI is a key thinking tool to examine investments in infrastructure, tools and automation. A realistic estimation of how much the tool will cost - in terms of development, integration, maintenance and fees versus the benefits we expect to gain, such as time saving, additional test coverage, will help us decide whether to “go for it” or not.

Some Critique of the Above

Essentially, all models are wrong, but some are useful - said George E. P. Box. This is true for the thought models that I just described. Since Testing is a learning activity, and learning is unpredictable, sometimes wandering around may be the right thing to do. Low risk and low ROI activity that we barely connect to the product quality in the first place, may lead us to critical information that has high ROI and high risk which is connected to the product quality.

What is your secret tool? Do you use a different version of similar tools? Let me know.

Professional meeting? JeST DO IT

This post is about my experience of organizing Tester’s meeting. It is true  as well for any other meeting, be it Software Developers, Social workers or a meeting of Puppeteers.

The 3rd meeting of JeST – the Jerusalem workshop on SW Testing is just over. Such a meeting is a great opportunity to meet fellow testers, and share experiences and ideas.

I am the proud founder of JeST, together with Shmuel Gershon, Asher Samuels and Alon Waisbard (look in Twitter for @sgershon, @absamuels  and @awaisbard ). I am telling you that because less than a year ago, organizing such a meeting, looked to me as a distant vision, something that will happen in the far future when I will develop the complex skills necessary to organize such an event.

But it doesn’t.

A moment during JeST2

Organizing a meeting is so easy, that I must share how easy it is, with you. The main issue to overcome is the mental block and just do it. To help you overcome this block, I prepared a list of the things that you don’t need, in order to hold such a meeting:

1)      A budget

You will be able to find a place to host a short meeting. You don’t need to rent a conference hall. It can be the room in the company of one of the meeting participants, or one’s home living room. Refreshments can be brought by the host or by the participants.  If it works for you and the potential participants, you can schedule a meeting in a quiet restaurant and address the need for both the location and refreshments.

2)      Sponsors

Since you don’t need a heavy budget, you don’t need an organization to support your meeting. Although I welcome cooperation and networking, which naturally happens in such meetings, I prefer to remain independent and stay away from organizations that try to use the meeting for their own promotion.

A thank you note should be enough as a return of the hosts good will.

3)      Heavy logistics

You don’t need a full-time administrator to organize such a meeting. Split the tasks between a few friends and it will not consume too much time. Use social networks to spread the word. Meetup service can simplify the process even more, in return for a payment of a few bucks.

(Late note: there are many Meetup groups which are allready dedicated to promote events like yours. See for example: ILTechTalks group. All you need is to suggest your event in such group, free of charge)

4)      Professional presenters and presentations

Not everyone who has interesting things to say is a professional presenter and the length of a

JeST3 agenda

presentation does not make it better (most times it’s the opposite). For our meetings, we chose a format of lightning talks followed by a discussion.  One time, we had a “regular” presentation, which was great too, but the most significant part for most of the participants was the discussions that we held.

5)      A large number of participants

Rating is overrated. While it is nice to see many people come to “your” meeting, a large audience has its drawbacks too – the atmosphere is less intimate, it’s harder to facilitate the discussion, and not every participant feels free to express his opinion. When I started to organize the first JeST meeting, I was worried that the meeting would not  draw a “respectful” number of participants.  I did the “Worst case scenario” drill and came to conclusion that if only me and the other 3 folks that shared the idea of organizing the meeting would come, it would not be a waste of time.  At the end the meeting turned out to be a success, also by the number of participants.

All the above are not a “must have” in order to organize your testers meeting. All you need is Testers and a love of testing.

A small cool macro that makes Mind Maps and Spreadsheets better friends

In case, like me, you belong to the Mind Maps lovers’ group, there’s good chance this tool will interest you.

I like Mind Maps because they are easy to create and evolve. They represent data in a way that makes sense to human beings.  

When you want to add a leaf to any branch of the data structure, you don’t need to mess around, as it is just a natural development of the idea representation.

On the other hand, Spreadsheets have their own advantages. They are able to perform calculations on the data, and sort and filter it.

I like to combine both Mind Maps and spreadsheets in my work. I summarize ideas in a Mind Map and move it to an Excel™ spreadsheet in order to use it in a way that involves calculations and filtering.

I use Xmind for creating Mind Maps. Moving the data from the Xmind application to an Excel sheet is very easy: copy-paste the central subject into the sheet. 

However, the data format in the target sheet is not very useful for my goal – each hierarchy is placed in a new column, as you can see in picture #1. 

I would be happier if I were able to have all the data in same column, indented by the hierarchy, as you can see in picture #2. 

It would be even cooler if we were able to use Excel’s “grouping” feature so we could see the exact hierarchic level of the data, see picture#3.

Pic#1

I will not keep secrets from you. I wrote an Excel VBA Macros that provides the wish list above.  Feel free to use it, just copy it into your VBA editor.

Note: while this tool is great for porting the Mind Map data into Excel, once you’ve used it, the data won’t be easily exported back to MindMap. If you find it necessary, you can create a VBA macro that will help to do that.

Pic#3

Pic#2

I created three macros: one that moves the data to one column and indents it according the hierarchy. The other groups the data according to the indentation and the third one which calls both macros.
Before that you run the macro, make sure that the cells that conatins the data are in the "selection" as you can see in the following video:

 If you have  any questions, Tweet me: @testermindset
Enjoy!
The VBA code:

Sub moveIndentGroup()

' This macro call the 2 other Macros in order to perform all actions usingcommand

    LastRow = Selection.Cells.Rows.Count

    Call moveAndIndent

    Call GroupIt(LastRow)

End Sub

Sub moveAndIndent()

 Dim rCell As Range

    Dim rRng As Range

    Set rRng = Selection

    For Each rCell In rRng.Cells

       If ((rCell <> "" Or rCell <> 0)) Then

        Cells(rCell.Row, 1).Value = rCell.Value

        Cells(rCell.Row, 1).IndentLevel = rCell.Column - 1

        If rCell.Column > 1 Then rCell.Value = ""

       End If

        Cells(rCell.Row, 1).HorizontalAlignment = xlLeft

    Next rCell

End Sub

Sub GroupIt(Optional LastRow)

    If IsMissing(LastRow) Then LastRow = Selection.Cells.Rows.Count

    For j = 1 To 5

        For i = 1 To LastRow

            If Cells(i, 1).IndentLevel = j Then

                FirstCell = i

                LastCell = i

                While (Cells(FirstCell, 1).IndentLevel <= Cells(LastCell + 1, 1).IndentLevel) And LastCell <= LastRow

                    LastCell = LastCell + 1

                Wend

                Range(Cells(FirstCell, 1), Cells(LastCell, 1)).Select

                Selection.Rows.Group

                i = LastCell + 1

            End If

        Next i

    Next j

End Sub

Dealing with Stress take 2

In case that you are reading this post, there is a chance that you read my blog post about Stress.

This post was transformed twice: first, I turned it into an article in Tea Time with testers magazine, than it became a presentation in QA&Test 2013 conference at Bilbao, Spain.

The conference was awesome. Hospitality was great. I had an opportunity to get to meet and share ideas

with many cool testers from all over the world. I was also able to experience presenting at an International conference.

Besides the format change – from a blog post to an article and then into a presentation, the ideas themselves emerged and developed as I got a lot of feedbacks while working on the material.

The main idea behind the work is the need to connect our Stress tests to the user’s needs . To do that, I suggest categorizing our Stress tests and failures into 3 main categories: Multiple experiments, Stability over time and Load.

While working on the presentation I added a few aspects which are connected to the failure classification and Stress test planning:

·         Assessment of risk when selecting risky flows for multiple experiments.

·         Taking in account the impact of the product on the system stability.

·         The need to find good oracles beyond the official requirements when defining the load and stability targets.

·         Perform load tests of few types :

o   The largest amount of data or actions which has meaning  for the users

o    The full capacity of the product – in order to spot degradation in the capacity before they has impact on the users.

·         Use good logging mechanism to gather data on all the experiments that your stress performs.

·         Monitor the system resources in order to quickly find stability issues .

Since I was scheduled to present on the last day of the conference, I had some time to get inspiration from a few people that I met during the 1^st two days. The night before the presentation, I changed the summary slide from a list into a mind map that summarizes my takes on the subject.

I am publishing the mind map and would like to ask you to review it and contribute to my initial work on that. I promise to give you credit if you’ll provide meaningful input.

Click to enlarge

Notes from the 1st conference at Ben-Gurion University of the Negev: Software Quality and Testing - Academia and Industry Get Together

I participated in the conference and would like to share my impressions and notes.

I really liked the idea of having such a conference, to get some exposure to what’s going on in the academic world regarding SW testing. This, together with promising talks from industry practitioners, was a good reason to head south and participate.
The hosting was great and the organizers deserve a good word for their efforts. The conference was held in the university “Senath auditorium”, which is a very pleasant and cozy (although well air conditioned) place. Everything was well organized.

The first talk, after a few greetings was by Dr. Jasbir Dhaliwal from the University of Memphis who talked about his experience in collaboration with the industry. He established the Systems Testing Excellence Program (STEP) in collaboration with Fedex.
He talked about the challenge of collaboration between the scientific approach of the academia and the art approach of the industry practitioners, which he called the Science – Art gap. 
During the day, several speakers referred to the different approaches between the industry and the academia, where the academia is focused on verification and the industry is more concerned with validation. 

Nir Caliv, a Principal Test Architect from Microsoft Israel, talked about Continuous Delivery and Test in Production. He talked about how the shift from producing boxed software packages to services in the cloud changed their development and testing methodology. For example, shifting from long development and test cycles that ends in mass distribution to very short ones that reach the production environment very fast.  He talked about the impact of the need and ability to change the SW quickly. He mentioned the move from modeling the system to using the “Real thing”, and from simulation of the end-user environment to monitoring of the production service itself. Monitoring gained a much more significant place than in the past and “Quality features” that enabled this monitoring were added to the product. The analysis of the monitored data become a major role of the “SW Development Engineer in Testing” as Microsoft calls their testers. They changed from passive monitoring to data mining that involves machine learning. 
It seems that when the monitoring capabilities role changed from being a “luxury” to a “necessity”, it forced them put a large effort on this area. This being done can teach organization whose product nature has not yet made the move to look at the benefits of this direction for their testing, and invest more in monitoring capabilities and gathering data from the production environment. 


In the lobby, Dr. Avi Ofer from ALM4U displayed a demo of a new method for verification using temporal logic . This method started with Hardware verification, but he shows how it can be used in Software verification, including the ability to go back in time and debug by replaying the actions without running the debugger again.

Dr. Meir Kalech from the Ben-Gurion University talked about “Artificial Intelligence Techniques to Improve Software Testing”. In his talk, he described a project that uses model based approach to diagnostics in complex environment like software, based on the IEEE Zoltar toolset for automatic fault localization. His system suggests scenarios for failure isolation. While he suggested that the system be used to suggest the “next step” to testers that experience a failure, my view is  that this is more suitable to be an addition to automation checks  than as a tool for human beings.

Ron Moussafi from Intel talked about "The Fab Experience: What Intel Fabs can teach us about Software”. Ron has a lot of experience in the semiconductors manufacturing world, but he is currently managing two test departments, one of them,  I work in. He talked about the difference in the approaches of the two industries. While the fab(Semiconductor fabrication plant) has a culture which focuses on quality, and quality is a main factor that is measured with no tolerance for incompatibility, the software industry culture is focused on other aspects. One of the causes of the difference is that in the manufacturing world the cost of one error is very noticeable, while in the SW industry, most of times the cost of error is hard to measure, but has the impact of “death by a 1000 papercuts”. While “Fab” people see themselves as scientists, the image of the SW guy is more of a “Lone ranger”. Ron suggested some actions to improve the quality culture of SW development organizations, like calculating and publishing the cost of defects and connecting between failure and the cause, in order to improve the process, instead on just focusing on fixing the problem.

Prof. Mark Last from the host University talked about Using Data Mining For Automated Design of Software Tests. He showed a method that does data fuzzing on a legacy “known to be good” software and uses data mining techniques in order to select a regression package and run it on a new version of the software.

Prof. Ron F. Kenet from KPA Ltd. talked about "Process Improvement and CMMI for Systems and Software". Since I am not really interested in the subject, I did not take notes on his talk. However, one of the examples he gave did grab my attention as it can be connected to one of the other talks. He demonstrated measurement of point of fault during the development process, which can be useful if you want to try Ron Moussafi’s suggestion to connect between failure and cause.

Dr. Amir Tomer from the Kinneret Academic College talked about "Software Intensive Systems Modeling - A Methodological Approach”. He described his method of using UML to describe SW systems. In his method, each entity has four characteristics: The ones that relate to the requirements from the entity: the environment it operates in and the services it provides, and ones that relate to the design: the structure and the behavior. He showed how he connects between the entities so the services and behavior of one entity, are actually the environment of the other entity.

Scott Barber is a well-known tester, especially if you are interested in performance testing. With no offence to the other speakers, he was the main reason for my participation in the conference.  Scott’s talk was "A Practitioner's View of Commercial Business's Desired Value Add from Software Testing”. He was able to educate not only the academic participants, but also the industry people on what is the view of the Commercial Business's decision makers of Testing and the challenges of explaining the added value of testing to them. This can be quite difficult when they think about testing as an additional thing they have to pay for having software built, like caffeine is needed, and don’t distinguish between Testing and quality. Scot’s said that “the industry pays for value to get to the market sooner, faster and cheaper”, these are the things that you want to focus on and make sure to communicate your contribution to their achievement.

I was not able to participate in the panel discussion afterwards “Product Quality and Software Quality:  Are They Aligned?”

To summarize, the conference was very interesting and exposed the participants to different views of Testing. While, as practitioner I was not familiar with the language and approach of the academic speakers and I have to admit that I giggled a bit when they talked about a “lines of code” or gave an example in FORTRAN, it was refreshing to hear such different views. 
The academic engagement with Testing is in its early phases which mean that it has lot of space for great things to happen. This conference was a big step in this direction and a good way to connect between people from the two disciplines. 

As an Intel employee, I am proud that Intel was a major sponsor of such a great event, which was open to anyone who has an interest in our field.

While the Verification aspects of the profession were well heard, I was missing academic speakers from other disciplines which have no less impact on our practice and I would argue that even more. Cooperation with researchers in the areas of Psychology, Anthropology, Education and more, may able the academia to address more of the industry’s needs and be more relevant to the industry.

Dealing with Stress

Stress failures and bug advocacy - looking at stress tests from a value perspective

Stress is part of your test strategy. You use it as a tool to test your product and find bugs. This is one of the “non-functional” test categories you run. Did you devote the time to think about what is actually being tested by your stress tests?

You may continue to test without answering this question, but when the time comes for bug advocacy, you have to defend your test strategy and findings, and this may force you to search for an answer.

What are we stressing for?

1)      Statistical failure  - Stress increases the chances of the appearance of a sporadic defect since it executes a flow a lot of times

2)      Run stability tests in a shorter time – the stress speeds up the time factor – failure reveals in a short time a defect that a system which runs in normal conditions (amount of data, number of simultaneous actions, etc.) will experience after a longer run.  A common example of such a failure is a memory leak found using the stress setup.

3)      Load (sometimes defined as a category by itself) – when we test how our system scales with multiple calls, large amount of data or both. Here, the failure reveals a point when the system fails to handle the load.

4)      Any  combination of 1, 2 or 3.

In a utopic scenario, when a stress related defect is reported, it follows the path of debug, root cause and fix. But in many cases, we will need our bug advocacy skills in order to convince our stakeholders of the need to fix the defect.

A typical bug discussion can start like this:

Developer: “Stress of 4½ hours and 5MB data files is not a normal usage of our system. A typical use case takes 15 minutes and a smaller amount of data. We should reject this bug.”

This point in the discussion can reveal whether you did your homework or not.

To decide that the failure is from the 1^st classification – statistical, we need to decompose the stress to a meaningful use case and run it over and over while bringing the system to a clean state between the each use case. Automation can be a big help here.

If we succeed in reproducing the failure under such conditions, our report will transform from a stress failure report to a use case failure report with reproduction rate. When we have a sufficient statistical sample, the impact is clear.

Pinpointing whether the failure is related to time or to load is more complex, as we need to “play” with both factors in order to reach a conclusion about the amount of time, load or both that is needed in order to cause the system to reach a failure point. The awareness of the possible options is an important tool in bug advocacy. For example, it can enhance stakeholder’s perspective when you are able to say that “we’re not sure yet, but it is possible that we will see the failure in normal conditions after a long period of time.”

Doing complete research before reporting the stress failure can consume lot of resources and time, so I don’t suggest delaying the report till the tester has all of the answers. Many times, we can reach faster and better conclusions about the failure from a focused code review or a debug log analysis.

I would like to suggest the following: learn to classify your stress failures.  When you see and report a stress failure, treat it as a start of the classification and investigation. While sometimes the report will be enough to call for a bug fix, many times it will serve as a call for investigation. During the investigation – make clear to stakeholders what you already know and what you don’t know yet. Make sure that new findings are updated in the bug and don’t be afraid to change the title to reflect it.

There is much more to learn than the basics I summarized in this post. Learning more about stress in general and about your specific system, can help you classify and investigate your stress failures and no less important – plan your stress tests better.

DOWNTIME NOTIFICATION

If you will it, it is no dream.

One day, the test engineers in the Testing lab came to work and went through their inbox. Among the mail was a short message from the Testing lab manager:

To: All testers

Subject: QC DOWNTIME NOTIFICATION: unlimited period

Dear testers, in order to try new ways of work, our test management system, Quality Center (AKA “QC”), will be down until further notice. There is no change in your task definitions, roles and responsibilities.

P.S. I will be out of office on vacation for the next two weeks.

At first glance, the testers were shocked, they re-read the message and moved their eyes to the message date, but it wasn’t April 1^st. They looked at the Hebrew calendar, but the month of Adar hadn’t started yet. As seasoned testers, they didn’t take the message as a fact and tried to enter the Quality Center site, without success. 

The coffee corner was crowded. Continuous humming of testers and team managers shaking their head from side to side in disbelief. Managers tried to reach the test lab manager via cell phone, SMS, email, Facebook, and twitter, but got no response. They approached the manager’s manager who told them: “Please talk with your test lab manager. I believe that it is important, but in order to change his decision you must contact him. Till then I will back his decision and won’t interfere.”

The most concerned was the project testing leadership team. They realized that they would not be able to query progress indicators and track whether they had reached coverage expectations for each work week: ww25 10%; ww26 20%; ww27 40%; ww28 60%; ww29 80%; ww30 90% and so on.

The Security team manager gave his team clear instructions: “Perform any legal or semi-legal operation to hack and bring up the system”. But even the efforts of the best white hat hackers team in the industry were not enough to bring up a system that was shutdown, power unplugged, behind a locked door.

Team leaders sat in the labs perplexed. Some asked their team members to perform some bug verifications and handle other minor issues, but didn’t have an idea as to what to do next. One of the leads had an idea: Shmuel, from the Sustaining product test team, told him once that he does not work with QC. As he knew Shmuel managed to perform testing, they could ask him for advice.

An expedition of several people approached Shmuel. The following dialogue took place:

Crowd: “Please help us. We need to run our cycle, but QC is down. How can we do it? Do we need some kind of magic?”

Shmuel: "Does QC run the cycle for you?"

Crowd: “No! But how will we know what tests to run?”

Shmuel: "Oh, knowing what tests to run is important, as we will want to look for information about what we don't know, and get a feeling for areas we consider at risk. Is that what you want from your tests too?"

Crowd: “Yes”

Shmuel: "So let's see: How do you decide what tests to run?"

Sara: "I go to QC and pull the list of tests that are not 'done'."

Shmuel: "We said we are looking for risk and new info – is that what you have in the not-’done’ list?"

Herzl: "Not exactly, it is a list of tests for old info."

Shmuel: "But your team does find new bugs using them, Herzl, how do you do that?"

Herzl: “I guess we find them because when we run the list of tests, we somehow know where things can show problems.”

Josh: “During test execution sometimes I see something behaves suspiciously. I investigate it and find a bug.”

Shmuel: "Josh, this is what happens in my team, too! But you said 'somehow', Herzl... how do you think you knew?"

Herzl: "I don't know... we've seen similar problems in the past?"

Shmuel: "Ah, so you use your experience as part of the secret. Do you have seasoned testers in your team?"

Herzl: “Yes."

Sara: "We also have new testers, they sometimes tests things we hadn't written or thought of."

Shmuel: "It looks like we let our teams think about risk and new information (what we said we were looking for) everyday! Is that so? Sara?"

Sara: "Well, yes... in a sense... for bugs."

Shmuel: "So what do you need QC for?"

Herzl: "How will they start if we don't give them a cycle?"

Shmuel: "How about you just tell them to start? You can ask them to list what they want to test without another list distracting them.”

Herzl: If you will it, it is no dream

Sara: “How would we make sure that teams distribute the work correctly and focus on the risk areas?”

Shmuel: “You do talk with your team members, don’t you? “

Sara (a bit offended): ”Sure, constantly!”

Shmuel: “When talking with your team you learn about what they tested, so you can take the opportunity to discuss focus areas and distribute the work.”

Sara: ”I guess this can be more meaningful than when I just hand them the test cycle and ask about progress.”

Josh: ”Without a list of test cases, how do you make sure that you will not forget to check important things or some details? Our product is pretty big and you can easily forget a part.”

Sara: ”I agree with Josh and want to add that sometimes we do find a bug when a test case fails.”

Shmuel: “Sara and Josh, this is a good question, and hard to answer. It seems that there’s an assumption that the central test list will solve the problem for you.

Testers work is solving this problem, and they have a wealth of tools for that – the test case repository is one of them. Renewing the test case list is another, which can be done by interviewing managers and programmers. Now you don’t have the central repository, but you can organize your own checklists and areas of coverage.”

Herzl: “Sound great, but are you able to report coverage and test results without having numbers of test cases and pass/fail results?”

Shmuel: “Let’s take a closer look at these results and see if we’ll miss them. Are found problems lost, unless there’s a failed test? We use a bug repository to log and study them, which is reviewed by the project managers.”

Herzl: “But without the pass results, do you suggest providing our message about coverage without supporting numbers?”

Shmuel: “Not at all! If you have supportive numbers you should provide them. But in your cycle case... if the receiver does not know what the tests are, there’s little support by counting them.“

Sara: “Well, all they ask is whether we will complete our work in time. And I think we will.“

The test team leaders start to disperse, each to his own team, to discuss the work without the test management database. In a few hours the whole testing department was back at work. Issues were raised, bugs were opened, and short meaningful coverage reports were provided.

Two weeks later. A short message appeared in the tester’s inbox:

To: All testers

Subject: COMPLETED: QC DOWNTIME NOTIFICATION:  server is up again

No one bothered to read the rest of the message.

Special thank to Shmuel Gershon for participating as a guest author in this post 

=====================================================================

Further reading:

Pradeep Soundararajan, Truth about test plan document & test case document

James Bach, , A Low Tech Testing Dashboard