Wednesday, July 28, 2010
The numbers monster
While writing about indicators and numbers, I remembered that James Bach described the moment when he realized that the test cases count he had been asked to give his manager was meaningless as a turning point in his career (utest blog, "Testing the limits" with James Bach).
smiling, I lean my head towards my laptop, taking a small nap.
Through the mist, an email message popped up to my face. The magical forest keeper forwarding me a message from the numbers monster: The numbers monster is hungry and wants me to give her some numbers.
I replied to the forest keeper: I understand that the numbers are very tasty, but they are complete junk food. We can ask the monster what is her purpose in digesting the numbers, so I will be able guide her to find some good herbs that will satisfy her, instead of fattening her with the junk food she asked for.
The forest keeper replied again in few seconds: The monster demand her numbers.
I replied back: our forests success depends on the monsters health. I know that she is used to the junk food that she gets from other creatures in the forest, but I don't think it is good for her. I watch monsters when they eat the numbers junk food, they start going in the wrong direction, and even when they go in the right direction, they waste the dwarfs energy supplying them junk food, instead of gardening the herb plot and taking care of the forest's prosperity.
The forest keeper replied again: The monster demands her numbers. As a forest keeper, I need you to co operate in order that the dwarfs will keep feeding the monsters numbers. It’s easy for you and for them, and most importantly, it gives the monsters the feeling that they control the forest and not the dwarfs.
My mind starts thinking how to convince the forest keeper to give up his tradition and do a better job for our forest, but then I woke up from the dream.
What do you do when you meet the forest keeper and the numbers monsters?
Wednesday, June 16, 2010
Humor and bug submission - comedy or tragedy?
Both the title and the description made me laugh, especially since I remember that many years ago, a few of my fellow testers and myself used to spend our lunch breaks playing Quake.
While enjoying the humor, all of my bells of bug submission guidelines rang. We know that bug submission is a very sensitive and responsible task, and the bug must be presented in an objective manner, which was not the case here. On the other hand, reading the bug gave me a real laugh, which we need while working so hard. In addition, I can even say that using the funny example helped me identify with the end-user described as not an experienced "Quake Tournament Player", since I suck at playing arcade Quake, too.
We know that humor should be used wisely - to relieve tension and give us a good laugh that relieves the stress. Even the serious discipline of medical treatment is aware of the contribution of humor and laugher to the patients and invented the profession of medical clown.
Should we go in the sterile “BKM” direction or share our sense of humor even in our bug descriptions?
Since I value risk taking, I would say that sometimes we can allow ourselves to go a bit wild with a few guidelines:
- Make sure you personally know the possible stakeholders who will look at, or work on, the bug, and make sure they know you and value a sense of humor.
- Don’t start with jokes before you've established your credibility as a serious bug submitter.
- Don’t do it too often. Once or twice in a project life cycle is enough.
- Make sure your joke is funny (perhaps you would like to test it before hand).
- No need to mention that the joke should not be offensive in any manner to anyone.
- At the end, make sure that the bug is perfect and clear from all other aspects. A bug with a joke, but without steps to reproduce, might look even worse than just a bad bug missing steps to reproduce.
- Understand that following all of the guidelines above cannot guarantee that you will not fall into hidden traps.
Comedy can easily turn into tragedy. Sometimes taking risks pays off, and sometimes not. Do it, but at your own risk!
Tuesday, April 27, 2010
Celebrating 10 years of testing - the Milestones and the People

I will follow the context driven school of testing, which emphasizes the people as the most important part of the Testing project and I will celebrate each milestone and mention the related people who helped me reach it.
Milestone: First mistake
Person: Myself (who else could I blame…?)
Bad decision – bad result – when I let development close a non-reproducible bug, it was a mistake - since such bugs tend to reproduce at a later stage... Fortunately, failure reproduced again before the product went out to certification, during the pre-certification process, but that caused a lot of noise. Believe me - learning this lesson made me a much better tester.
Milestone: Leading a team of really great testers
People: Jose & Itzik
Jose, a new student (now a firmware architect) and Itzik, a tester with some experience (now a SW developer) joined my team. Together we built a team of passionate testers who challenged the tested SW. Without knowledge of too many methodologies and processes we explored and invented new methods and tools. We did a great job for our stakeholders.
I discovered my skill as a team leader – not intimidated by the talents of others – but helping the team express their abilities.
Milestone: My really great bug
Person: Jose
When I wondered “what would happen if…” and did the extra action that exposed a major failure in the product, I did not recognize how great a bug it was. Only when Jose (appearing in this post for the 2nd time), pointed out the thought process and mindset that led me to find this bug did I understand the significance of this occasion. Since then, a framed hard copy of the bug entry hangs in my cubicle and I refer to it as my own Tester Certification.
Milestone: Redefining myself as a Tester
Person: James Marcus Bach
When I was sent to his “How to be an expert tester” session, I did not anticipate its impact. Perhaps I was ready after 5 years of experimenting and perhaps it was James’ style, but this session made me start a journey of reading, talking and thinking about testing, exposing myself to different approaches, finding my preferences and defining my approach to different aspects of our profession.
Milestone: Participating in the Methodologies workgroup
People: The workgroup members
Participating and contributing to the methodologies workgroup of my testing department was not only an honor, but a challenge. Besides reviewing the work of others, I contributed two of the chapters of our methodology document. I had to explore, define, write and discuss the methodology of bug submission and test case authoring, exposing myself to criticism and debate.
Milestone: Starting to blog
People: Shmuel Gershon, Pradeep Soundararajan, Parimala Shankaraiah
During the last year, I started blogging. The inspiration came from my friend Shmuel Gershon who has been blogging about testing for a few years. I started blogging with a first post that reviewed Test books and magazines. Then I read Pradeep Soundararajan’s post “Why good software testers should come out of the well?”, and started blogging my own ideas too. I had been thinking about it for a long time, but his last post gave me the final push after a few months of an idle blog. Parimala Shankaraiah with her refreshing blog “Curious Tester” inspired me by showing how simple ideas turn into brilliant and refreshing posts.
All of these people and milestones have made me the Tester that I am today, and made for a thoroughly enjoyable decade. Thank you, And have a great next decade.
Monday, March 29, 2010
Bugs in children’s literature
Katz’s painting on the book cover is also the most famous kid’s literature painting bug.
The book starts with the following words: "בְּעֵמֶק יְפֵה בֵּין כְּרָמִים וְשָׂדוֹת עוֹמֵד מִגְדָּל בֶּן חָמֵשׁ קוֹמוֹת", or in my free english translation "In a beautiful valley between vineyards and fields stands a five floor tower".

If you are in the business of Testing and Development that sounds familiar, no?
Even when the author himself is the illustrator bugs slip through the paintbrush.
The great Israeli Children books author and illustrator of the current century (And I can say that as a father of three who has been reading bed time stories for few years), Rinat Hoffer (No Wikipedia article about her yet) does it too. In her funny book “Tom’s day”,about a day's in a nutty girl's life, she asks:”What's Grandpa hiding in his coat? A toy car? Or a lollipop? No! today he brings Grandma Rachel!” Very funny, and when you open the coat you can actualy see Grandma Rachel hiding, but also a lollipop, which we just said is not there... My son spots the bug easily, and when we will have time we might complain to Rinat.
These are only two examples. From time to time me and my kids spot more bugs in our books. You are welcome to submit your examples in this posts' comments.
The thing that most annoys me when spotting such bugs is that I suspect that most of the time the bugs were spotted before the publishing of the books, but the editors did not have enough respect for our kids intelligence to think that they would notice the mismatches.
The analogy for SW development is obvious. Who didn’t hear “Who will ever use this feature?” or “No one will notice?”. We would better remember Katz’s lesson and respect the anonymous user.
Monday, February 8, 2010
Methods for testing in a scripted test design environment
The original title of the post was "Methods for exploration in the scripted enviroment". However I went a step forward, using Michael Boloton's distinction, to sharpen the point of the discussion. I know that not all of my points in this post adhere to Michael's definitions in his article (for example about using specs), but I permit myself to use his most basic definitions for the sake of my post.
Michael Bolton in his famous post about Testing Vs. Checking defines:
Checking Is Confirmation
Checking is something that we do with the motivation of confirming existing beliefs. Checking is a process of confirmation, verification, and validation. When we already believe something to be true, we verify our belief by checking…. Checking is focused on making sure that the program doesn't fail.
Testing Is Exploration and Learning
Testing is something that we do with the motivation of finding new information. Testing is a process of exploration, discovery, investigation, and learning. When we configure, operate, and observe a product with the intention of evaluating it, or with the intention of recognizing a problem that we hadn't anticipated, we're testing. We're testing when we're trying to find out about the extents and limitations of the product and its design, and when we're largely driven by questions that haven't been answered or even asked before…
Now that we have refreshed our minds with Bolton’s definition of Testing, the challenge of doing testing in a scripted environment is clearer. In other words – when we are in a scripted environment that naturally emphasizes following the written instructions which is actually checking, how do we encourage the testers to do testing when scripted test design is a given? This is the case in most of the testing that I am involved in, although ET is getting more and more space in the activities that I am part of.
I would like to list a few methods that I used, learned or was witness to. I don't have a “preferred one”, but I like them all:
"Run, then read” –Read the test title, design the test in your mind or on a piece of paper, run the test, then read the test steps and complete any items you hadn't thought of, if there are such items.
“Do something new” – In this method test results of “Pass” or “Fail” are not enough. The test report tool also has a mandatory field of a description of an additional test that you perfomed.
“Reveal the sources” – the test case author attached, linked or copied the requirements and/or design document as a relevant part of the test case. When the base of the test design appears in front of the tester as part of the test design itself – the tester is invited to again review the logic behind the design and to think about new ideas.
“Dialogue with the script” - Use the script as an exploration tool - see what the test case isn’t telling me, and to do so in each of its steps. “Do A, B, C; expect D” becomes: “Does it matter where A started from? What happens to B if A isn’t done? In how many ways I can do C? What else happens to the system apart from D?
“Doubt everything” or “Challenge everything” - start the tests considering the Test Case as wrong, equivocating and lying; everything is disputed (sometimes aggressively). The script told us to do something, we didn’t agree with it, or with the design that generated it, or with the requirement that generated the design, or even with the customer who generated the requirement.
(Thanks to my colleagues: BatSheva, Itzik, Shmuel and Jose who inspired me with these and other methods during the last decade.)
Enviromental factors that increase Testing among testers:
Focus on bugs, less on coverage: A culture of “Fail is good”, example - “Bug of the week” discussion in each team meeting...
Emphasize the responsibility of the testers for the quality, not for the test execution
Encourage continuous learning - reading the requirements(again…), learning the product technologies and continuous discussions with the customers (and their rep. in the organization) architects and developers .
Are you (still) working in a scripted environment? How do you increase testing? please Tell me.
Tuesday, November 3, 2009
Read the requirements (again)!
עֲשֵׂה לְךָ תֵּבַת עֲצֵי-גֹפֶר, קִנִּים תַּעֲשֶׂה אֶת-הַתֵּבָה; וְכָפַרְתָּ אֹתָהּ מִבַּיִת וּמִחוּץ, בַּכֹּפֶר. טו וְזֶה, אֲשֶׁר תַּעֲשֶׂה אֹתָהּ: שְׁלֹשׁ מֵאוֹת אַמָּה, אֹרֶךְ הַתֵּבָה, חֲמִשִּׁים אַמָּה רָחְבָּהּ, וּשְׁלֹשִׁים אַמָּה קוֹמָתָהּ. טז צֹהַר תַּעֲשֶׂה לַתֵּבָה, וְאֶל-אַמָּה תְּכַלֶּנָּה מִלְמַעְלָה, וּפֶתַח הַתֵּבָה, בְּצִדָּהּ תָּשִׂים; תַּחְתִּיִּם שְׁנִיִּם וּשְׁלִשִׁים, תַּעֲשֶׂהָ. יז וַאֲנִי, הִנְנִי מֵבִיא אֶת-הַמַּבּוּל מַיִם עַל-הָאָרֶץ, לְשַׁחֵת כָּל-בָּשָׂר אֲשֶׁר-בּוֹ רוּחַ חַיִּים, מִתַּחַת הַשָּׁמָיִם: כֹּל אֲשֶׁר-בָּאָרֶץ, יִגְוָע. יח וַהֲקִמֹתִי אֶת-בְּרִיתִי, אִתָּךְ; וּבָאתָ, אֶל-הַתֵּבָה--אַתָּה, וּבָנֶיךָ וְאִשְׁתְּךָ וּנְשֵׁי-בָנֶיךָ אִתָּךְ. יט וּמִכָּל-הָחַי מִכָּל-בָּשָׂר שְׁנַיִם מִכֹּל, תָּבִיא אֶל-הַתֵּבָה--לְהַחֲיֹת אִתָּךְ: זָכָר וּנְקֵבָה, יִהְיוּ. כ מֵהָעוֹף לְמִינֵהוּ, וּמִן-הַבְּהֵמָה לְמִינָהּ, מִכֹּל רֶמֶשׂ הָאֲדָמָה, לְמִינֵהוּ--שְׁנַיִם מִכֹּל יָבֹאוּ אֵלֶיךָ, לְהַחֲיוֹת. כא וְאַתָּה קַח-לְךָ, מִכָּל-מַאֲכָל אֲשֶׁר יֵאָכֵל, וְאָסַפְתָּ, אֵלֶיךָ; וְהָיָה לְךָ וְלָהֶם, לְאָכְלָה. כב וַיַּעַשׂ, נֹחַ: כְּכֹל אֲשֶׁר צִוָּה אֹתוֹ, אֱלֹהִים--כֵּן עָשָׂה. וַיֹּאמֶר יְהוָה לְנֹחַ, בֹּא-אַתָּה וְכָל-בֵּיתְךָ אֶל-הַתֵּבָה: כִּי-אֹתְךָ רָאִיתִי צַדִּיק לְפָנַי, בַּדּוֹר הַזֶּה. ב מִכֹּל הַבְּהֵמָה הַטְּהוֹרָה, תִּקַּח-לְךָ שִׁבְעָה שִׁבְעָה--אִישׁ וְאִשְׁתּוֹ; וּמִן-הַבְּהֵמָה אֲשֶׁר לֹא טְהֹרָה הִוא, שְׁנַיִם--אִישׁ וְאִשְׁתּוֹ. ג גַּם מֵעוֹף הַשָּׁמַיִם שִׁבְעָה שִׁבְעָה, זָכָר וּנְקֵבָה, לְחַיּוֹת זֶרַע, עַל-פְּנֵי כָל-הָאָרֶץ
Harel demonstrates great requirements reading skills. I am not sure what is the chance that my son will be a tester in his future (currently he wants to have a career as a gaming SW tester, which in his eyes is the perfect profession), but for me this experience is an additional reminder of the danger of executing a test from a test plan without reading again the requirements.
The Ark model.
Wednesday, October 21, 2009
Observations, Art, practicing and inspiration from other blogs
Recently, I have read few blogs that introduce the idea of practicing in describing art in order to improve the skill of description.
Describing what we see is essential part when we test complicated systems. Beyond being accurate, objective and concise, you need to take the system generalist role - be able to conclude.
Parimala Shankaraiah talks in her refreshing blog "curios tester" about painting the picture - she describes how she addressed the challenge in Marlena Compton blog post “Visionary Testing” . I am quoting Marlena:
I challenge you to find a work of art be it a painting, sculpture, installation or anything you deem “art-worthy” and study it. This can be in a museum, a coffee house or your mom’s living room. Once you feel you have an understanding of what you are looking at, try to communicate your understanding with words.
Not only that I decided, like Parimala to address it and post my notes in my blog, but I also told Parimala that I want to do that. She replied to me that I’ll let her know how I did this exercise, so now I am obligated to try this :-). Hopefully you will see that in few lines.
Another interesting thing was that Parimala wrote about different perceptions of different people, and that Interpretation requires more knowledge about the context.
We could see a deomonstration of of that if we will look at the example that Marlena gave in her post. she shows a picture named “Esther before Ahasuerus”. She describes Esther (the women in the picture) as “another chick in a dress who was fairly bitter about life in general”.
I guess that when Parimala(She is an Indian women) read that post, she might agree or not, but when I did, that description looked so out of the context. I also saw a women and a king, but reading the picture name rang all of my bells. As an Israeli and Jewish, I have heard the story behind the scene so many times, so for me Esther is not a “bitter chick” but a great women in a middle of a brave act that saved our people from destruction. Eventually, that gave us a great holiday - Purim, that we celebrate every year, having lot of fun.
As for the exercise of picking and describing a picture?
I picked the “The Raft of the Medusa” (1819) by Theodore Gericault. I saw the picture around 13-14 years ago, when I visited the museum Deorsey in Paris. I remember that I was very impressed, but don’t recall any specific observations when I saw it. Forgive me if it will be too depressing, but this is a really impressive piece of art. Let's go to my lab notes:

What I saw?
A raft full of survivors from a ship sink.
Survivors pack fills the raft with desperate and terrified people. exception for that is the black man that waves towards the horizon
the view behind is depressing dark skies with clouds that promise a storm that could sink the raft
What I inferred?
Depression. The end is coming. The black waving man looks like an exception that should be examined carefully.
What I saw?
On the lower left side, fainted, almost dead bodies. Near them a desperate man sitting hopeless perhaps mourning for them
On the upper right side, near the mast, few terrified people, probably seeing their death in their eyes. Going right – we could see the black man waving. His look is hidden, but we could tell that he is looking forward something (rescue? Freedom?). beneath him another man, bent is waving to the same direction few other figure point to that direction too
What I inferred?
The situation puts the different people in different states: some are lost their consciousness, some are mourning, some are terrified and some are hoping for rescue
What I saw?
Examining the different figures one by one, There is a gradual progress from one side of the people that lies down to the black man which stands up and waving. Some of the figures shows both the “hope” and the “terrified” situation
What I inferred?
A fragile situation, that moves like a raft, between desperation and hope.

