Dealing with Stress

Stress failures and bug advocacy - looking at stress tests from a value perspective

Stress is part of your test strategy. You use it as a tool to test your product and find bugs. This is one of the “non-functional” test categories you run. Did you devote the time to think about what is actually being tested by your stress tests?

You may continue to test without answering this question, but when the time comes for bug advocacy, you have to defend your test strategy and findings, and this may force you to search for an answer.

What are we stressing for?

1)      Statistical failure  - Stress increases the chances of the appearance of a sporadic defect since it executes a flow a lot of times

2)      Run stability tests in a shorter time – the stress speeds up the time factor – failure reveals in a short time a defect that a system which runs in normal conditions (amount of data, number of simultaneous actions, etc.) will experience after a longer run.  A common example of such a failure is a memory leak found using the stress setup.

3)      Load (sometimes defined as a category by itself) – when we test how our system scales with multiple calls, large amount of data or both. Here, the failure reveals a point when the system fails to handle the load.

4)      Any  combination of 1, 2 or 3.

In a utopic scenario, when a stress related defect is reported, it follows the path of debug, root cause and fix. But in many cases, we will need our bug advocacy skills in order to convince our stakeholders of the need to fix the defect.

A typical bug discussion can start like this:

Developer: “Stress of 4½ hours and 5MB data files is not a normal usage of our system. A typical use case takes 15 minutes and a smaller amount of data. We should reject this bug.”

This point in the discussion can reveal whether you did your homework or not.

To decide that the failure is from the 1^st classification – statistical, we need to decompose the stress to a meaningful use case and run it over and over while bringing the system to a clean state between the each use case. Automation can be a big help here.

If we succeed in reproducing the failure under such conditions, our report will transform from a stress failure report to a use case failure report with reproduction rate. When we have a sufficient statistical sample, the impact is clear.

Pinpointing whether the failure is related to time or to load is more complex, as we need to “play” with both factors in order to reach a conclusion about the amount of time, load or both that is needed in order to cause the system to reach a failure point. The awareness of the possible options is an important tool in bug advocacy. For example, it can enhance stakeholder’s perspective when you are able to say that “we’re not sure yet, but it is possible that we will see the failure in normal conditions after a long period of time.”

Doing complete research before reporting the stress failure can consume lot of resources and time, so I don’t suggest delaying the report till the tester has all of the answers. Many times, we can reach faster and better conclusions about the failure from a focused code review or a debug log analysis.

I would like to suggest the following: learn to classify your stress failures.  When you see and report a stress failure, treat it as a start of the classification and investigation. While sometimes the report will be enough to call for a bug fix, many times it will serve as a call for investigation. During the investigation – make clear to stakeholders what you already know and what you don’t know yet. Make sure that new findings are updated in the bug and don’t be afraid to change the title to reflect it.

There is much more to learn than the basics I summarized in this post. Learning more about stress in general and about your specific system, can help you classify and investigate your stress failures and no less important – plan your stress tests better.

When you Feel Rejected…

It is common to see a bug rejected as “Not a requirement”. It sometimes hurts as it pushes aside your valuable feedback with a process related excuse.

Common examples are

·         When a requirement includes implementation details and the devil (our bug) is in those details – the bug is actually in the requirements.

·         When an issue is detected by using an oracle other than the official requirement (for example one of the HICCUPPS heuristics).

Some less logical examples that I’ve actually seen:

·         When the fix involves someone who is not committed to the effort yet – for example when a Platform bug requires a Software workaround, especially if the effort is big. “Not a requirement” here actually means “Not my responsibility”.

·         When a bug is the result of a design limitation. “Not a requirement” here is actually “It’s not my fault, it’s the Designers fault” and many times the “Bug fix is too expensive”.

Choose the playing field according to the context.

There’s a big field of product value that includes a smaller field of the requirements scope. I play in both. When in order to find this disputable bug, we kicked the ball to the big field, when someone moved its status to “Not a requirement”, he kicked the ball to a smaller field.

Now it’s your turn to select your move according to the context:

1)      Accept the bug rejection

Sometimes the other side of the coins’ argument has validity.

2)      Kick the ball within the requirements field

While the “requirements – yes or no?” argument limits the discussion, if you are able to win it, it will be easier to lead the bug to fix, as the bug handling process is usually more efficient and faster than the requirements definition and approval process. Beware of being too persuasive and winning the argument without a proper reviewer.

3)      Kick the ball to the big field of value again

When the rejection is correct process-wise but not product value wise, it’s time to play in the big field with the big boys. Advocate your bug to stakeholders and decision makers, learn more from customer support and architects or submit a requirement change request. Running in this field is long distance, scoring a goal is much rarer, but this is where you will meet the professional players and improve your own skills.

While the requirements discussion can be more or less relevant, playing beyond it might bring the best rewards.

Humor and bug submission - comedy or tragedy?

One of the testers working on the validation project I am leading, submitted a bug with the following title: "Unbelievably short period of appearance of the feature menu - feature impossible to use". In addition he mentions in the bug description that the feature is impossible to enter for anyone but an experienced "Quake Tournament Player".

Both the title and the description made me laugh, especially since I remember that many years ago, a few of my fellow testers and myself used to spend our lunch breaks playing Quake.

While enjoying the humor, all of my bells of bug submission guidelines rang. We know that bug submission is a very sensitive and responsible task, and the bug must be presented in an objective manner, which was not the case here. On the other hand, reading the bug gave me a real laugh, which we need while working so hard. In addition, I can even say that using the funny example helped me identify with the end-user described as not an experienced "Quake Tournament Player", since I suck at playing arcade Quake, too.

We know that humor should be used wisely - to relieve tension and give us a good laugh that relieves the stress. Even the serious discipline of medical treatment is aware of the contribution of humor and laugher to the patients and invented the profession of medical clown.

Should we go in the sterile “BKM” direction or share our sense of humor even in our bug descriptions?

Since I value risk taking, I would say that sometimes we can allow ourselves to go a bit wild with a few guidelines:

• Make sure you personally know the possible stakeholders who will look at, or work on, the bug, and make sure they know you and value a sense of humor.
• Don’t start with jokes before you've established your credibility as a serious bug submitter.
• Don’t do it too often. Once or twice in a project life cycle is enough.
• Make sure your joke is funny (perhaps you would like to test it before hand).
• No need to mention that the joke should not be offensive in any manner to anyone.
• At the end, make sure that the bug is perfect and clear from all other aspects. A bug with a joke, but without steps to reproduce, might look even worse than just a bad bug missing steps to reproduce.
• Understand that following all of the guidelines above cannot guarantee that you will not fall into hidden traps.

Comedy can easily turn into tragedy. Sometimes taking risks pays off, and sometimes not. Do it, but at your own risk!

Celebrating 10 years of testing - the Milestones and the People

This month marks 10 years since I moved from the I.T. support field into SW testing. Celebrations often find you very busy with other activities like I am now these days, in a middle of a validation project, but nevertheless I would like to give some thought to important milestones during this testing decade.

I will follow the context driven school of testing, which emphasizes the people as the most important part of the Testing project and I will celebrate each milestone and mention the related people who helped me reach it.

Milestone: First mistake
Person: Myself (who else could I blame…?)
Bad decision – bad result – when I let development close a non-reproducible bug, it was a mistake - since such bugs tend to reproduce at a later stage... Fortunately, failure reproduced again before the product went out to certification, during the pre-certification process, but that caused a lot of noise. Believe me - learning this lesson made me a much better tester.

Milestone: Leading a team of really great testers
People: Jose & Itzik
Jose, a new student (now a firmware architect) and Itzik, a tester with some experience (now a SW developer) joined my team. Together we built a team of passionate testers who challenged the tested SW. Without knowledge of too many methodologies and processes we explored and invented new methods and tools. We did a great job for our stakeholders.
I discovered my skill as a team leader – not intimidated by the talents of others – but helping the team express their abilities.

Milestone: My really great bug
Person: Jose
When I wondered “what would happen if…” and did the extra action that exposed a major failure in the product, I did not recognize how great a bug it was. Only when Jose (appearing in this post for the 2nd time), pointed out the thought process and mindset that led me to find this bug did I understand the significance of this occasion. Since then, a framed hard copy of the bug entry hangs in my cubicle and I refer to it as my own Tester Certification.

Milestone: Redefining myself as a Tester
Person: James Marcus Bach
When I was sent to his “How to be an expert tester” session, I did not anticipate its impact. Perhaps I was ready after 5 years of experimenting and perhaps it was James’ style, but this session made me start a journey of reading, talking and thinking about testing, exposing myself to different approaches, finding my preferences and defining my approach to different aspects of our profession.

Milestone: Participating in the Methodologies workgroup
People: The workgroup members
Participating and contributing to the methodologies workgroup of my testing department was not only an honor, but a challenge. Besides reviewing the work of others, I contributed two of the chapters of our methodology document. I had to explore, define, write and discuss the methodology of bug submission and test case authoring, exposing myself to criticism and debate.

Milestone: Starting to blog
People: Shmuel Gershon, Pradeep Soundararajan, Parimala Shankaraiah
During the last year, I started blogging. The inspiration came from my friend Shmuel Gershon who has been blogging about testing for a few years. I started blogging with a first post that reviewed Test books and magazines. Then I read Pradeep Soundararajan’s post “Why good software testers should come out of the well?”, and started blogging my own ideas too. I had been thinking about it for a long time, but his last post gave me the final push after a few months of an idle blog. Parimala Shankaraiah with her refreshing blog “Curious Tester” inspired me by showing how simple ideas turn into brilliant and refreshing posts.

All of these people and milestones have made me the Tester that I am today, and made for a thoroughly enjoyable decade. Thank you, And have a great next decade.

Bugs in children’s literature

During the last week, we heard that the famous books illustrator Shmuel Katz (see also Hebrew Wikipedia article ) passed away. His biography as a holocaust survivor, Kibbutz member and an extreme left-wing activist may be very interesting, but he is most famous for illustrating the children’s book Dira Lehaskir ("Apartment for Rent") by Lea Goldberg which was written in 1959 and became one of the classics of Hebrew children's literature.


Katz’s painting on the book cover is also the most famous kid’s literature painting bug.
The book starts with the following words: "בְּעֵמֶק יְפֵה בֵּין כְּרָמִים וְשָׂדוֹת עוֹמֵד מִגְדָּל בֶּן חָמֵשׁ קוֹמוֹת", or in my free english translation "In a beautiful valley between vineyards and fields stands a five floor tower".

When Katz passed away, one of the T.V. channels broadcast a recent interview with him. In the interview he admits that he regrets the moment when he noticed that he painted only four floors(As you can see in the picture above) while there are five in the text. He didn’t take the time to fix his mistake since he believed that no one would notice. (Un)fortunely, the book become a best seller and he got many complaints about this bug for the rest of his life.

If you are in the business of Testing and Development that sounds familiar, no?

Even when the author himself is the illustrator bugs slip through the paintbrush.
The great Israeli Children books author and illustrator of the current century (And I can say that as a father of three who has been reading bed time stories for few years), Rinat Hoffer (No Wikipedia article about her yet) does it too. In her funny book “Tom’s day”,about a day's in a nutty girl's life, she asks:”What's Grandpa hiding in his coat? A toy car? Or a lollipop? No! today he brings Grandma Rachel!” Very funny, and when you open the coat you can actualy see Grandma Rachel hiding, but also a lollipop, which we just said is not there... My son spots the bug easily, and when we will have time we might complain to Rinat.

These are only two examples. From time to time me and my kids spot more bugs in our books. You are welcome to submit your examples in this posts' comments.

The thing that most annoys me when spotting such bugs is that I suspect that most of the time the bugs were spotted before the publishing of the books, but the editors did not have enough respect for our kids intelligence to think that they would notice the mismatches.

The analogy for SW development is obvious. Who didn’t hear “Who will ever use this feature?” or “No one will notice?”. We would better remember Katz’s lesson and respect the anonymous user.