Tester Mindset

When you Feel Rejected…

2011-11-17T19:29:00.001+02:00

It is common to see a bug rejected as “Not a requirement”. It sometimes hurts as it pushes aside your valuable feedback with a process related excuse.

Common examples are

· When a requirement includes implementation details and the devil (our bug) is in those details – the bug is actually in the requirements.

· When an issue is detected by using an oracle other than the official requirement (for example one of the HICCUPPS heuristics).

Some less logical examples that I’ve actually seen:

· When the fix involves someone who is not committed to the effort yet – for example when a Platform bug requires a Software workaround, especially if the effort is big. “Not a requirement” here actually means “Not my responsibility”.

· When a bug is the result of a design limitation. “Not a requirement” here is actually “It’s not my fault, it’s the Designers fault” and many times the “Bug fix is too expensive”.

Choose the playing field according to the context.

There’s a big field of product value that includes a smaller field of the requirements scope. I play in both. When in order to find this disputable bug, we kicked the ball to the big field, when someone moved its status to “Not a requirement”, he kicked the ball to a smaller field.

Now it’s your turn to select your move according to the context:

1) Accept the bug rejection

Sometimes the other side of the coins’ argument has validity.

2) Kick the ball within the requirements field

While the “requirements – yes or no?” argument limits the discussion, if you are able to win it, it will be easier to lead the bug to fix, as the bug handling process is usually more efficient and faster than the requirements definition and approval process. Beware of being too persuasive and winning the argument without a proper reviewer.

3) Kick the ball to the big field of value again

When the rejection is correct process-wise but not product value wise, it’s time to play in the big field with the big boys. Advocate your bug to stakeholders and decision makers, learn more from customer support and architects or submit a requirement change request. Running in this field is long distance, scoring a goal is much rarer, but this is where you will meet the professional players and improve your own skills.

While the requirements discussion can be more or less relevant, playing beyond it might bring the best rewards.

The Double Sin of the Early Perfect Test Case

2011-07-24T22:25:00.000+03:00

Since I started leading my current testing team, I’ve been struggling with the test case base.

There are a few factors that made the test case base clumsy and outdated. One of the most stunning facts for me was that even test cases that demonstrated big investment in details were often out dated. Despite the large investment, some details were wrong, had changed, had never been true, or were outdated. Often, you could find new testers struggling to understand and execute the test baseline.

The First Sin: Detailed Gen 0 Test Cases

In my experience when test cases are created before the test designer sees and experiences the product, it’s more than likely that they will not be accurate.

The reason for the failure is the limitation of our mind to perfectly imagine an abstract design. Sometimes even the designers doesn’t have a 100% complete design. While you can plan many things ahead of time, you can also anticipate that you will have gaps in your planning, but not anticipate their exact location.

The Second Sin: Detailed Gen 1 Test Cases

What about tests that successfully made it from Gen 0 to Gen 1 and proved to be correct? What about tests that were designed after the product was introduced and tried? They might not suffer from the first sin, but they will suffer from the second sin. Although these tests were accurate in the assumptions about the product itself, not all of them were the correct ones to run. Moreover, some of the tests that did a great job for Gen 1, finished their duty. Using these tests in regression will not be efficient.

As we progress with the test execution, we learn more about the risks of the product. At the end of the first generation testing we can plan better regression testing for the next generations. Typically we will add a small number of test cases and get rid of a larger amount of tests.

Conclusion: investing in too many detailed test cases during Gen0 and Gen1 is not efficient.

I’ll try to define basic guidelines to deal with this issue:

1) Lower the expectations from Gen0 and Gen 1 test cases – understand the built-in limitation of these test cases: Gen 0 might be inaccurate and Gen 1 will not fit your regression needs.

2) Seek for alternatives when planning Gen 0 and Gen 1 test cases. For example, use checklists instead of steps (See “The Value of Checklists and the Danger of Scripts” by Cem Kaner).

3) Try to thinks of better uses of your time during test planning. For example, invest in automation infrastructure during preparation.

4) Realize that moving from Gen 1 o Gen 2 will require more time in test documentation, and is not just copy-paste from Gen0-1 test cases. In this stage, you can save time by creating less “perfect” test cases of new features in the same product introduced at this time.

5) Consider the possibility that you will come to like your lean Gen 0 and Gen 1 test cases so much, that you won’t want to invest in more details for the regression test case base.

In case you claim that your experience is different and it’s possible to create perfect re-usable test plans in early stages, I can think of the following possibilities:

1) You are a better product and test designer than the ones I work with (please mentor me).

2) You don’t have complex and innovative products like the products that I test.

3) You follow a perfect process that prevents you from falling in such traps (and I would like to hear more about it).

Is there a Pesticide paradox in testing?

2011-05-08T14:30:00.000+03:00

A B-B-Q of one term (I am using the term BBQ in the spirit of Israel's Yom Ha'atzmaut :-) )

As a tester, I have heard and read the term “Pesticide paradox “ on many occasions . However, I do not feel comfortable with it so I avoid using it. In the last few days, I decided to examine it more carefully – does this term makes sense? I did some googling to explore the common use of the term in SW testing, the definition of the paradox in the real pesticide world, and to try to give an answer to the question.

The original paradox is explained in Wikipedia :
“The Paradox of the pesticides is a paradox that states that by applying pesticide to a pest, one may in fact increase its abundance. This happens when the pesticide upsets natural predator-prey dynamics in the ecosystem.” I'll refer to this definition as the "original".

The common use of the term in Testing as I experienced it, is to describe that when using scripted tests (automated in most cases), which are repeated over and over again, eventually the same set of test cases will no longer find any new defects (I took a quote from the ITCQB syllabus which is great source to find "terms of common use"). I'll refer to this definition as the "common use"

I also found the explanation that "A static set of tests will become less effective as developers learn to avoid making mistakes that trigger the tests." (A paper by Rex Black ).

I could say that the common use of the term is to describe that repeating the same checks tends to yield less bugs from run to run. I agree that usually this is true – according to my experience, since bugs get fixed, and usually when no major changes are introduced and no very bad development occurs, products become more stable from release to release also, the development learning explanation is logical.

If we try to correlate between the Testing common use of the term and the original one, we will be able to see a very loose connection – SW bugs do not increase due to the fact that you repeat some checks, and moreover, where is the paradox here? If you ask a question over and over, it is very likely that most of times you'll get the correct answer. I don’t call this a paradox.

When you analyze a term, it is good practice to read the source. I on't have the book Software testing techniques by Boris Beizer, from which is the term origin, but thanks to http://www.softwarequotes.com/ , I found it:
First law: The pesticide paradox. Every method you use to prevent or find bugs leaves a residue of subtler bugs against which those methods are ineffective.
- Boris Beizer - Chapter 1, Section 1.7. Boris notes that farmers solve this problem by planting sacrifice crops for the bugs to eat, and laments that programmers are unable to write sacrifice functions., Software testing techniques by Boris Beizer , ISBN: 0442206720. I'll refer to this quote as "Bezier's".

Well, that makes sense too, and is a good foundation law before you learns about methods – any method is not fully effective. Like with the common use term, I don't see the paradox.

I'll summarize my conclusions on the subject

• The connection between the original term – the biological phenomena of the "Pesticide paradox" and the common use in the testing world is mostly due to the use of the term “bug” to describe a defect, and that the original paradox deals with type of in efficiency when trying to pesticide pests.

• A clear logical paradox appear in the original phenomena – you kill bugs, but this increases their abundance, while the Bezier's and the common use of the term talk about less efficiency, not a paradox. A possible response to this statement will be to argue that when you do something and it is not efficient this is a paradox, to my taste this is too apologetic argument.

• The original SW testing usage quote from Bezier is a warning about relying on a sole method, while the common use by others, which usually refer to Bezier as the source, is to describe the decreased efficiency of repeating a scripted test.

It’s fine to use a cool term with loose analogy to describe your idea, but as you can see in our example, this might cause others to "steal" your term to describe other things (and worse – reference you as the source). In addition, it will be hard to convince people with critical thinking to use your term. I will leave the pesticide paradox to its original meaning.

Note about terminology

2011-01-04T18:14:00.000+02:00

Sometimes, It’s all about branding. When you want to sell a product or an approach, using terminology that will "sell" your approach to your stakeholders or the professional community has impact on the chances that it will be accepted .
When Fred Hoyle coined the term Big Bang during a 1949 BBC radio broadcast, he did not anticipate that he is doing a branding service to the competitive theory. According to Hoyle, who favored an alternative "steady state" cosmological model, he used the striking image to highlight the difference between the two models. Probably he did it too well :-)

Markus Gärtner in his post Active vs. passive testing introduce refreshing terminology for what we use to call Testing Vs. Checking or Exploratory Vs. Scripted. He uses the terms Active Vs. Passive testing. He also talk about the role of judgment which is part of being active, but basically, the new pair of active Vs. Passive comes to describe Research, critical, exploratory approach versos executing the planned tests, checking and following defined scripts.

This new terminology has some benefits on the terms we are used to. It's not using a term that we already use to describe wider area like "Testing". And unlike the term exploratory it's not suffer from the "unstructured" public image.

Using the definition of quality as a tool for context awareness

2010-11-11T17:00:00.000+02:00

What is quality?
Since testers job is to perform quality assessments, ask questions and provide answers regarding quality, understanding what quality is can help identify dilemmas in our work and put them in context.
My preferred definition for quality is “value to someone” (Jerry Weinberg). Cem Kaner adds the extension “who matters”. Usually I refer to this VIP as a “User”.

Recently, I noticed that using this definition helps identify context and explain the context to others during discussion.
A few examples:

Focusing a discussion on the goal rather than the process.
Process is important, but sometimes process discussions disconnect from the goal. For example, when a bug is opened and there is a discussion about whether it’s a requirement violation or not, providing insight on the User value could help direct the discussion to a productive place. This is also true when a tester spots an issue and is not sure whether it falls under his responsibility to report it – “is there a threat to the value to the users?” is a good litmus test to aid the decision.

Selecting a process
When defining a process, understanding how it’s connected to the value to the user is a good way to examine it. When we define a process it should connect our efforts to the goal rather than disconnect them. A negative example is when mixing between the priority for fixing the bug and the bug Severity. Many times there is correlation between the two, but it will be a good idea to define process that will address the cases when there is a difference between the two (like setting different field for each goal) so the information of the value threat severity will not be replaced by the work plan.

Determine classification of a problem
We face many types of problems. Some related to the quality of the product and some interfere with other aspects of our work, delay our progress or block our testing efforts. When a testability issue is examined in perspective of user value, it can be underrated since our inability to test efficiently is the real issue here, and not the impact of the tested attribute on the end user.
Sometimes there are two types of issues combined together in one problem description. Distinguishing between the value to the user and the other issue, helps provide a clearer explanation.

Overcoming the tunnel effect when setting bug severity
Setting correct bug exposure classification helps the bug life cycle start on the right foot. When a tester tests his area of responsibility and spots a problem, sometimes it’s not easy to relate to the big picture – what will be the impact on the user? Will he be able to recover? Or in other words – what is the threat in terms of value to the user. Answering this question easily directs the bug submitter to specify the correct bug severity.

Indicators of Useful or Useless Indicators

2010-09-02T15:49:00.000+03:00

The Testing Experience magazine published my article about Indicators of Useful or Useless Indicators In its recent issue about metrics.

I am very happy with this article publication for the following reasons:

I was able to transform my thoughts that came from my experience, into a context-driven article.
The publication exposed more of my thoughts and writing to my colleagues within the design center.

Following is an embedded version of the article. I recommend to download the magazine issue itself .

Wishing you a Happy Hebrew New Year שנה טובה ומתוקה,
Issi

Peaks are fun, but bit scary

2010-08-01T14:43:00.000+03:00

Writer Meta Thoughts.
Last week was definitely a peak in my writing journey. On Sunday, an article of mine was accepted for print by "Testing Experience" magazine. On Tuesday, Pradeep Soundararajan mentioned my name among other "Good thinkers and future experts". I wrote a post on Wednesday, and on Thursday discovered that my site had a record number of visitors. A short investigation led me to the traffic source - The following Tweet by James Bach:
jamesmarcusbach: Another new sapient testing blogger (avoids pat answers, critical self-analysis). I like this guy http://bit.ly/dl042u
Thank you, James!
What a week! No wonder, this week was also my Birthday week. While enjoying the compliments, a concern sneaks into mind - How do I maintain my reputation? The concern fades when I look down the path - remembering the journey I started a year and a half ago, daring to blog my first post and going from there, searching for my inner voice and improving my writing skills. Continuing the journey, looks less scary than taking the first step.
Here's to another year of new Insights and Peaks.

The numbers monster

2010-07-28T08:35:00.000+03:00

I just finished polishing my article about meaningful indicators; I had hit the "Send" button and felt good. I was able to express my thoughts about meaningful and meaningless numbers, and my article had been accepted to print.

While writing about indicators and numbers, I remembered that James Bach described the moment when he realized that the test cases count he had been asked to give his manager was meaningless as a turning point in his career (utest blog, "Testing the limits" with James Bach).

smiling, I lean my head towards my laptop, taking a small nap.
Through the mist, an email message popped up to my face. The magical forest keeper forwarding me a message from the numbers monster: The numbers monster is hungry and wants me to give her some numbers.

I replied to the forest keeper: I understand that the numbers are very tasty, but they are complete junk food. We can ask the monster what is her purpose in digesting the numbers, so I will be able guide her to find some good herbs that will satisfy her, instead of fattening her with the junk food she asked for.

The forest keeper replied again in few seconds: The monster demand her numbers.

I replied back: our forests success depends on the monsters health. I know that she is used to the junk food that she gets from other creatures in the forest, but I don't think it is good for her. I watch monsters when they eat the numbers junk food, they start going in the wrong direction, and even when they go in the right direction, they waste the dwarfs energy supplying them junk food, instead of gardening the herb plot and taking care of the forest's prosperity.

The forest keeper replied again: The monster demands her numbers. As a forest keeper, I need you to co operate in order that the dwarfs will keep feeding the monsters numbers. It’s easy for you and for them, and most importantly, it gives the monsters the feeling that they control the forest and not the dwarfs.

My mind starts thinking how to convince the forest keeper to give up his tradition and do a better job for our forest, but then I woke up from the dream.

What do you do when you meet the forest keeper and the numbers monsters?

Humor and bug submission - comedy or tragedy?

2010-06-16T21:26:00.000+03:00

One of the testers working on the validation project I am leading, submitted a bug with the following title: "Unbelievably short period of appearance of the feature menu - feature impossible to use". In addition he mentions in the bug description that the feature is impossible to enter for anyone but an experienced "Quake Tournament Player".

Both the title and the description made me laugh, especially since I remember that many years ago, a few of my fellow testers and myself used to spend our lunch breaks playing Quake.

While enjoying the humor, all of my bells of bug submission guidelines rang. We know that bug submission is a very sensitive and responsible task, and the bug must be presented in an objective manner, which was not the case here. On the other hand, reading the bug gave me a real laugh, which we need while working so hard. In addition, I can even say that using the funny example helped me identify with the end-user described as not an experienced "Quake Tournament Player", since I suck at playing arcade Quake, too.

We know that humor should be used wisely - to relieve tension and give us a good laugh that relieves the stress. Even the serious discipline of medical treatment is aware of the contribution of humor and laugher to the patients and invented the profession of medical clown.

Should we go in the sterile “BKM” direction or share our sense of humor even in our bug descriptions?

Since I value risk taking, I would say that sometimes we can allow ourselves to go a bit wild with a few guidelines:

Make sure you personally know the possible stakeholders who will look at, or work on, the bug, and make sure they know you and value a sense of humor.
Don’t start with jokes before you've established your credibility as a serious bug submitter.
Don’t do it too often. Once or twice in a project life cycle is enough.
Make sure your joke is funny (perhaps you would like to test it before hand).
No need to mention that the joke should not be offensive in any manner to anyone.
At the end, make sure that the bug is perfect and clear from all other aspects. A bug with a joke, but without steps to reproduce, might look even worse than just a bad bug missing steps to reproduce.
Understand that following all of the guidelines above cannot guarantee that you will not fall into hidden traps.

Comedy can easily turn into tragedy. Sometimes taking risks pays off, and sometimes not. Do it, but at your own risk!

Celebrating 10 years of testing - the Milestones and the People

2010-04-27T22:35:00.000+03:00

This month marks 10 years since I moved from the I.T. support field into SW testing. Celebrations often find you very busy with other activities like I am now these days, in a middle of a validation project, but nevertheless I would like to give some thought to important milestones during this testing decade.

I will follow the context driven school of testing, which emphasizes the people as the most important part of the Testing project and I will celebrate each milestone and mention the related people who helped me reach it.

Milestone: First mistake
Person: Myself (who else could I blame…?)
Bad decision – bad result – when I let development close a non-reproducible bug, it was a mistake - since such bugs tend to reproduce at a later stage... Fortunately, failure reproduced again before the product went out to certification, during the pre-certification process, but that caused a lot of noise. Believe me - learning this lesson made me a much better tester.

Milestone: Leading a team of really great testers
People: Jose & Itzik
Jose, a new student (now a firmware architect) and Itzik, a tester with some experience (now a SW developer) joined my team. Together we built a team of passionate testers who challenged the tested SW. Without knowledge of too many methodologies and processes we explored and invented new methods and tools. We did a great job for our stakeholders.
I discovered my skill as a team leader – not intimidated by the talents of others – but helping the team express their abilities.

Milestone: My really great bug
Person: Jose
When I wondered “what would happen if…” and did the extra action that exposed a major failure in the product, I did not recognize how great a bug it was. Only when Jose (appearing in this post for the 2nd time), pointed out the thought process and mindset that led me to find this bug did I understand the significance of this occasion. Since then, a framed hard copy of the bug entry hangs in my cubicle and I refer to it as my own Tester Certification.

Milestone: Redefining myself as a Tester
Person: James Marcus Bach
When I was sent to his “How to be an expert tester” session, I did not anticipate its impact. Perhaps I was ready after 5 years of experimenting and perhaps it was James’ style, but this session made me start a journey of reading, talking and thinking about testing, exposing myself to different approaches, finding my preferences and defining my approach to different aspects of our profession.

Milestone: Participating in the Methodologies workgroup
People: The workgroup members
Participating and contributing to the methodologies workgroup of my testing department was not only an honor, but a challenge. Besides reviewing the work of others, I contributed two of the chapters of our methodology document. I had to explore, define, write and discuss the methodology of bug submission and test case authoring, exposing myself to criticism and debate.

Milestone: Starting to blog
People: Shmuel Gershon, Pradeep Soundararajan, Parimala Shankaraiah
During the last year, I started blogging. The inspiration came from my friend Shmuel Gershon who has been blogging about testing for a few years. I started blogging with a first post that reviewed Test books and magazines. Then I read Pradeep Soundararajan’s post “Why good software testers should come out of the well?”, and started blogging my own ideas too. I had been thinking about it for a long time, but his last post gave me the final push after a few months of an idle blog. Parimala Shankaraiah with her refreshing blog “Curious Tester” inspired me by showing how simple ideas turn into brilliant and refreshing posts.

All of these people and milestones have made me the Tester that I am today, and made for a thoroughly enjoyable decade. Thank you, And have a great next decade.

Bugs in children’s literature

2010-03-29T15:02:00.000+03:00

During the last week, we heard that the famous books illustrator Shmuel Katz (see also Hebrew Wikipedia article ) passed away. His biography as a holocaust survivor, Kibbutz member and an extreme left-wing activist may be very interesting, but he is most famous for illustrating the children’s book Dira Lehaskir ("Apartment for Rent") by Lea Goldberg which was written in 1959 and became one of the classics of Hebrew children's literature.

Katz’s painting on the book cover is also the most famous kid’s literature painting bug.
The book starts with the following words: "בְּעֵמֶק יְפֵה בֵּין כְּרָמִים וְשָׂדוֹת עוֹמֵד מִגְדָּל בֶּן חָמֵשׁ קוֹמוֹת", or in my free english translation "In a beautiful valley between vineyards and fields stands a five floor tower".

When Katz passed away, one of the T.V. channels broadcast a recent interview with him. In the interview he admits that he regrets the moment when he noticed that he painted only four floors(As you can see in the picture above) while there are five in the text. He didn’t take the time to fix his mistake since he believed that no one would notice. (Un)fortunely, the book become a best seller and he got many complaints about this bug for the rest of his life.

If you are in the business of Testing and Development that sounds familiar, no?

Even when the author himself is the illustrator bugs slip through the paintbrush.
The great Israeli Children books author and illustrator of the current century (And I can say that as a father of three who has been reading bed time stories for few years), Rinat Hoffer (No Wikipedia article about her yet) does it too. In her funny book “Tom’s day”,about a day's in a nutty girl's life, she asks:”What's Grandpa hiding in his coat? A toy car? Or a lollipop? No! today he brings Grandma Rachel!” Very funny, and when you open the coat you can actualy see Grandma Rachel hiding, but also a lollipop, which we just said is not there... My son spots the bug easily, and when we will have time we might complain to Rinat.

These are only two examples. From time to time me and my kids spot more bugs in our books. You are welcome to submit your examples in this posts' comments.

The thing that most annoys me when spotting such bugs is that I suspect that most of the time the bugs were spotted before the publishing of the books, but the editors did not have enough respect for our kids intelligence to think that they would notice the mismatches.

The analogy for SW development is obvious. Who didn’t hear “Who will ever use this feature?” or “No one will notice?”. We would better remember Katz’s lesson and respect the anonymous user.

Methods for testing in a scripted test design environment

2010-02-08T23:15:00.000+02:00

The original title of the post was "Methods for exploration in the scripted enviroment". However I went a step forward, using Michael Boloton's distinction, to sharpen the point of the discussion. I know that not all of my points in this post adhere to Michael's definitions in his article (for example about using specs), but I permit myself to use his most basic definitions for the sake of my post.

Michael Bolton in his famous post about Testing Vs. Checking defines:
Checking Is Confirmation
Checking is something that we do with the motivation of confirming existing beliefs. Checking is a process of confirmation, verification, and validation. When we already believe something to be true, we verify our belief by checking…. Checking is focused on making sure that the program doesn't fail.

Testing Is Exploration and Learning
Testing is something that we do with the motivation of finding new information. Testing is a process of exploration, discovery, investigation, and learning. When we configure, operate, and observe a product with the intention of evaluating it, or with the intention of recognizing a problem that we hadn't anticipated, we're testing. We're testing when we're trying to find out about the extents and limitations of the product and its design, and when we're largely driven by questions that haven't been answered or even asked before…

Now that we have refreshed our minds with Bolton’s definition of Testing, the challenge of doing testing in a scripted environment is clearer. In other words – when we are in a scripted environment that naturally emphasizes following the written instructions which is actually checking, how do we encourage the testers to do testing when scripted test design is a given? This is the case in most of the testing that I am involved in, although ET is getting more and more space in the activities that I am part of.

I would like to list a few methods that I used, learned or was witness to. I don't have a “preferred one”, but I like them all:

"Run, then read” –Read the test title, design the test in your mind or on a piece of paper, run the test, then read the test steps and complete any items you hadn't thought of, if there are such items.

“Do something new” – In this method test results of “Pass” or “Fail” are not enough. The test report tool also has a mandatory field of a description of an additional test that you perfomed.

“Reveal the sources” – the test case author attached, linked or copied the requirements and/or design document as a relevant part of the test case. When the base of the test design appears in front of the tester as part of the test design itself – the tester is invited to again review the logic behind the design and to think about new ideas.

“Dialogue with the script” - Use the script as an exploration tool - see what the test case isn’t telling me, and to do so in each of its steps. “Do A, B, C; expect D” becomes: “Does it matter where A started from? What happens to B if A isn’t done? In how many ways I can do C? What else happens to the system apart from D?

“Doubt everything” or “Challenge everything” - start the tests considering the Test Case as wrong, equivocating and lying; everything is disputed (sometimes aggressively). The script told us to do something, we didn’t agree with it, or with the design that generated it, or with the requirement that generated the design, or even with the customer who generated the requirement.

(Thanks to my colleagues: BatSheva, Itzik, Shmuel and Jose who inspired me with these and other methods during the last decade.)

Enviromental factors that increase Testing among testers:

Focus on bugs, less on coverage: A culture of “Fail is good”, example - “Bug of the week” discussion in each team meeting...

Emphasize the responsibility of the testers for the quality, not for the test execution

Encourage continuous learning - reading the requirements(again…), learning the product technologies and continuous discussions with the customers (and their rep. in the organization) architects and developers .

Are you (still) working in a scripted environment? How do you increase testing? please Tell me.

Read the requirements (again)!

2009-11-03T21:58:00.000+02:00

This week I helped my 7 years old Son, Harel, in a school project – building a model of Noah’s Ark. Since I liked it to be educational and to teach him a bit about approaching a project, I asked him to list all the Ark's requirements directly from the source. We opened the Bible and read (I am quoting here the original Hebrew text and the American standard version, Genesis 6,13):

עֲשֵׂה לְךָ תֵּבַת עֲצֵי-גֹפֶר, קִנִּים תַּעֲשֶׂה אֶת-הַתֵּבָה; וְכָפַרְתָּ אֹתָהּ מִבַּיִת וּמִחוּץ, בַּכֹּפֶר. טו וְזֶה, אֲשֶׁר תַּעֲשֶׂה אֹתָהּ: שְׁלֹשׁ מֵאוֹת אַמָּה, אֹרֶךְ הַתֵּבָה, חֲמִשִּׁים אַמָּה רָחְבָּהּ, וּשְׁלֹשִׁים אַמָּה קוֹמָתָהּ. טז צֹהַר תַּעֲשֶׂה לַתֵּבָה, וְאֶל-אַמָּה תְּכַלֶּנָּה מִלְמַעְלָה, וּפֶתַח הַתֵּבָה, בְּצִדָּהּ תָּשִׂים; תַּחְתִּיִּם שְׁנִיִּם וּשְׁלִשִׁים, תַּעֲשֶׂהָ. יז וַאֲנִי, הִנְנִי מֵבִיא אֶת-הַמַּבּוּל מַיִם עַל-הָאָרֶץ, לְשַׁחֵת כָּל-בָּשָׂר אֲשֶׁר-בּוֹ רוּחַ חַיִּים, מִתַּחַת הַשָּׁמָיִם: כֹּל אֲשֶׁר-בָּאָרֶץ, יִגְוָע. יח וַהֲקִמֹתִי אֶת-בְּרִיתִי, אִתָּךְ; וּבָאתָ, אֶל-הַתֵּבָה--אַתָּה, וּבָנֶיךָ וְאִשְׁתְּךָ וּנְשֵׁי-בָנֶיךָ אִתָּךְ. יט וּמִכָּל-הָחַי מִכָּל-בָּשָׂר שְׁנַיִם מִכֹּל, תָּבִיא אֶל-הַתֵּבָה--לְהַחֲיֹת אִתָּךְ: זָכָר וּנְקֵבָה, יִהְיוּ. כ מֵהָעוֹף לְמִינֵהוּ, וּמִן-הַבְּהֵמָה לְמִינָהּ, מִכֹּל רֶמֶשׂ הָאֲדָמָה, לְמִינֵהוּ--שְׁנַיִם מִכֹּל יָבֹאוּ אֵלֶיךָ, לְהַחֲיוֹת. כא וְאַתָּה קַח-לְךָ, מִכָּל-מַאֲכָל אֲשֶׁר יֵאָכֵל, וְאָסַפְתָּ, אֵלֶיךָ; וְהָיָה לְךָ וְלָהֶם, לְאָכְלָה. כב וַיַּעַשׂ, נֹחַ: כְּכֹל אֲשֶׁר צִוָּה אֹתוֹ, אֱלֹהִים--כֵּן עָשָׂה. וַיֹּאמֶר יְהוָה לְנֹחַ, בֹּא-אַתָּה וְכָל-בֵּיתְךָ אֶל-הַתֵּבָה: כִּי-אֹתְךָ רָאִיתִי צַדִּיק לְפָנַי, בַּדּוֹר הַזֶּה. ב מִכֹּל הַבְּהֵמָה הַטְּהוֹרָה, תִּקַּח-לְךָ שִׁבְעָה שִׁבְעָה--אִישׁ וְאִשְׁתּוֹ; וּמִן-הַבְּהֵמָה אֲשֶׁר לֹא טְהֹרָה הִוא, שְׁנַיִם--אִישׁ וְאִשְׁתּוֹ. ג גַּם מֵעוֹף הַשָּׁמַיִם שִׁבְעָה שִׁבְעָה, זָכָר וּנְקֵבָה, לְחַיּוֹת זֶרַע, עַל-פְּנֵי כָל-הָאָרֶץ

Make thee an ark of gopher wood; rooms shalt thou make in the ark, and shalt pitch it within and without with pitch. And this is how thou shalt make it: the length of the ark three hundred cubits, the breadth of it fifty cubits, and the height of it thirty cubits. light shalt thou make to the ark, and to a cubit shalt thou finish it upward; and the door of the ark shalt thou set in the side thereof; with lower, second, and third stories shalt thou make it. And I, behold, I do bring the flood of waters upon this earth, to destroy all flesh, wherein is the breath of life, from under heaven; everything that is in the earth shall die. But I will establish my covenant with thee; and thou shalt come into the ark, thou, and thy sons, and thy wife, and thy sons wives with thee. And of every living thing of all flesh, two of every sort shalt thou bring into the ark, to keep them alive with thee; they shall be male and female. Of the birds after their kind, and of the cattle after their kind, of every creeping thing of the ground after its kind, two of every sort shall come unto thee, to keep them alive. And take thou unto thee of all food that is eaten, and gather it to thee; and it shall be for food for thee, and for them. Thus did Noah; according to all that God commanded him, so did he. And Jehovah said unto Noah, Come thou and all thy house into the ark; for thee have I seen righteous before me in this generation. Of every clean beast thou shalt take to thee seven and seven, the male and his female; and of the beasts that are not clean two, the male and his female."

So Harel took a pencil and listed the requirements of his model in his own words:

1. It should be made pales from a certain wood type

2. The ark should be sealed (with pitch)

3. Size in cubits will be 300(Length) X 50(width) X 30(height)

4. A window should be on the upper floor in size of one cubit

5. A door should be on the side of the Ark

6. The Ark should have 3 floors

7. In addition: The Ark will hold Noah’s family and the beasts: a pair of each kind, but seven of each “clean” kind.

When he was done listing, we noticed a few things which are not usually shown in the typical drawings or models of this favorite subject: I never saw anyone painting the Ark in black, however, it’s very likely that after Noah pitched the Ark from outside and inside he didn’t paint it brown and draw wood parts on it, like is shown in most pictures. No one also painted seven sheep or giraffes although these animals are from the “clean” type according to the tradition and Noah might have brought seven of them. We also noticed that the triangle roof and the ramp are optional components. Then we went to work – fulfilling the requirements without the bias we had from other sources, and learned a lesson about getting the requirements from the original source.

Harel demonstrates great requirements reading skills. I am not sure what is the chance that my son will be a tester in his future (currently he wants to have a career as a gaming SW tester, which in his eyes is the perfect profession), but for me this experience is an additional reminder of the danger of executing a test from a test plan without reading again the requirements.

The traditional test execution process (unlike ET), is built as a linear line of of turning the requirments and SW design into test cases and then executing them, but our mind is not linear. The ideas and understanding of the product evolve with time and experience, and our original perspective changes - for better, since it's more mature with time, and for worse, since it's likly to disconnect from the original sources of knowledge. Returning to the requirements at each stage of the execution will reconnect us to our mission definition and will give us an opportunity to relearn the material when our mind is more ready.

My recommendation to everyone who opens a bug or executes a test is always: Read the requirements (again)!

The Ark model.

We didn’t paint it black (we thought brown is nicer). We placed two

sheep and not seven (resources limitation), but we did it consciously.

Observations, Art, practicing and inspiration from other blogs

2009-10-21T23:11:00.000+02:00

Practicing in the art of description.

Recently, I have read few blogs that introduce the idea of practicing in describing art in order to improve the skill of description.

Describing what we see is essential part when we test complicated systems. Beyond being accurate, objective and concise, you need to take the system generalist role - be able to conclude.

Parimala Shankaraiah talks in her refreshing blog "curios tester" about painting the picture - she describes how she addressed the challenge in Marlena Compton blog post “Visionary Testing” . I am quoting Marlena:

I challenge you to find a work of art be it a painting, sculpture, installation or anything you deem “art-worthy” and study it. This can be in a museum, a coffee house or your mom’s living room. Once you feel you have an understanding of what you are looking at, try to communicate your understanding with words.

Not only that I decided, like Parimala to address it and post my notes in my blog, but I also told Parimala that I want to do that. She replied to me that I’ll let her know how I did this exercise, so now I am obligated to try this :-). Hopefully you will see that in few lines.

Another interesting thing was that Parimala wrote about different perceptions of different people, and that Interpretation requires more knowledge about the context.
We could see a deomonstration of of that if we will look at the example that Marlena gave in her post. she shows a picture named “Esther before Ahasuerus”. She describes Esther (the women in the picture) as “another chick in a dress who was fairly bitter about life in general”.

I guess that when Parimala(She is an Indian women) read that post, she might agree or not, but when I did, that description looked so out of the context. I also saw a women and a king, but reading the picture name rang all of my bells. As an Israeli and Jewish, I have heard the story behind the scene so many times, so for me Esther is not a “bitter chick” but a great women in a middle of a brave act that saved our people from destruction. Eventually, that gave us a great holiday - Purim, that we celebrate every year, having lot of fun.

As for the exercise of picking and describing a picture?

I picked the “The Raft of the Medusa” (1819) by Theodore Gericault. I saw the picture around 13-14 years ago, when I visited the museum Deorsey in Paris. I remember that I was very impressed, but don’t recall any specific observations when I saw it. Forgive me if it will be too depressing, but this is a really impressive piece of art. Let's go to my lab notes:

What I saw?

A raft full of survivors from a ship sink.
Survivors pack fills the raft with desperate and terrified people. exception for that is the black man that waves towards the horizon
the view behind is depressing dark skies with clouds that promise a storm that could sink the raft

What I inferred?
Depression. The end is coming. The black waving man looks like an exception that should be examined carefully.

What I saw?
On the lower left side, fainted, almost dead bodies. Near them a desperate man sitting hopeless perhaps mourning for them
On the upper right side, near the mast, few terrified people, probably seeing their death in their eyes. Going right – we could see the black man waving. His look is hidden, but we could tell that he is looking forward something (rescue? Freedom?). beneath him another man, bent is waving to the same direction few other figure point to that direction too

What I inferred?
The situation puts the different people in different states: some are lost their consciousness, some are mourning, some are terrified and some are hoping for rescue

What I saw?
Examining the different figures one by one, There is a gradual progress from one side of the people that lies down to the black man which stands up and waving. Some of the figures shows both the “hope” and the “terrified” situation

What I inferred?
A fragile situation, that moves like a raft, between desperation and hope.

Create your tester portfolio

2009-09-22T21:25:00.000+03:00

The following presentation that I have created with my friend Shmuel Gershon, suggest ways for novice testers to gain experience by doing open source and crowd source testing and. We did not invent the wheel here, but we discuss and give some guiding that could be useful for novice testers.
This presentation was a result of a request that we got from few Tech career SQA course graduates, for some training that will increase their value in the eyes of potential employers. they suggested that we will teach them the ITCQB certification exam. however, we thought that we could do something more meaningful by giving them this workshop.

link to our work:

Create Your Tester Portfolio

View more presentations from sgershon.

and as a shared document in scribd

The Beauty of technique and the bias Beast

2009-09-17T17:20:00.000+03:00

Recently, I see around me more emphasis on test design techniques. People get trained in common techniques, and I hear voices that call for more documentation of the techniques used in test design documents.
I would like to have a closer look on the trend by analyzing a simple but extreme example to demonstrate the hidden pitfalls that could be found in this path.

Let's take a look at the lovely pair of equivalent partitioning and Boundary values analysis. Both are usually the first most basic techniques to be mentioned when testing design techniques are discussed.

Equivalent partitioning method is derived from the fact that testing all possible values is not possible, and that one representative value from each partition is sufficient in order to have good enough testing.
Boundary values analysis come to compensate the weakness of Equivalent partitioning. Although that the specification does not specify a different behavior for processing boundary values, we do know that this areas are likely to fail due to architecture or programming bugs.

Let's apply the techniques for deriving data for our test design. Assume that we have to test the following function:

MemoryCopy(Source,Length,Destination)

The requirments specification says:

This function is used by software other components to copy memory cells, given the source memory cell number, the length of data to copy and the destination cell to copy the cells group to.

The environment of operation of the function is a memory area consists of 0 up to N memory cells that could contain any data.

Using the equivalent partitioning approach, we will analyze our environment: We have 3 data partitions: below Zero, Zero up to N and N up to infinite. Each one of our parameters could get one of them. Using pair wise planning will lower the amount of tested combinations so to get good enough coverage for the function will run the following test cases and call it TDD#1:

Case#	Source	Length	Destination
1	0-N	0-N	0-N
2	N+1-infinite	0-N	0-N
3	0-N	Negative	0-N
4	0-N	N+1-infinite	0-N
5	0-N	0-N	Negative
6	0-N	0-N	N+1-infinite
7	Negative	0-N	0-N

Note, that in this case, only one out the seven cases is a positive test case (Case#1), since the specification defines only one valid partition.

Since we already know that we must not use this technique without the complementary boundary values, we will specify the boundary values that we must check: instead of using 0-N we will use 0, 1,N-1 and N which are the boundary values within the valid partition range, -1 and N+1 as the boundary values of the invalid partitions. Thanks to all pairs algorithm, we end up with the following 44 cases(The negative ones are marked with ~) that we will call TDD#2:

Case#	Source	Length	Destination	Case#	Source	Length	Destination
1	N-1	1	0	12	1	N-1	N-1
2	N	N	N-1	13	N-1	0	N-1
3	N	0	N	14	1	0	0
4	0	1	N-1	15	0	0	1
5	N	N-1	0	16	0	N	0
6	N-1	N-1	1	17	1	N	1
7	0	N-1	N	18	~-1	N	0
8	1	1	1	19	N	N	~-1
9	N-1	N	N	20	N-1	~-1	N-1
10	N	1	1	21	~-1	0	1
11	1	1	N	22	0	N	~N+1

Case#	Source	Length	Destination	Case#	Source	Length	Destination
23	N	~-1	1	34	~-1	N	N
24	0	~-1	0	35	N-1	0	~-1
25	1	~-1	N	36	~N+1	0	N
26	~N+1	N-1	N-1	37	0	N-1	~-1
27	N-1	~N+1	N	38	N	1	~-1
28	~N+1	1	0	39	1	1	~-1
29	0	~N+1	1	40	~N+1	N-1	1
30	~-1	1	N-1	41	~N+1	N	0
31	N-1	1	~N+1	42	1	~N+1	0
32	N	~N+1	N-1	43	N	0	~N+1
33	1	N-1	~N+1	44	~-1	N-1	N

So far so good, but we just been trapped…

While in terms of valid, invalid data ranges and their boundaries we did state of the art coverage, we forgot the main functional attributes of our function while it copies data from source to destination, and has to maintain data integrity.

In order to look up what we missed, let's go back to our tester natural mind set and do analysis without bounding ourselves to technique.

Getting the mention above specification on the first time, we will use our drawing board to draw a typical environment, and use our best test technique abstraction ability to imagine the possible cases and list them. We'll give attention to the partitioning and the boundaries when they will come in our path, but we will defer the full discussion of them to a later stage, so to remain focus on the functionality.

Figure - drawing board

Having the drawing in front of us, we would imagine and note the different possible cases:

Let's move memory cells 0 -3 to cells N-3 up to N. Now, intuitively, we will move them to N-2 (cases 1 and 2).

But even nicer would be to copy cells 0 - 3 to cells 1-4 since potential data loss could be here when some of the destination data overlaps the source data. In this point we'll take the idea further and imagine how will the function will copy the data (cases 3 and 4).

In case that data is copied cell by cell, it should better take rightmost cell (in our example "D" from cell 3) and copy it to the rightmost cell of the destination (cell 4) and so on till it reaches the leftmost cell, so to avoid loss of data. From the other hand if we will try to copy cells 1-4 to cells 0-3, the function must start with the leftmost cell to avoid the same data loss.

Let's sum our findings into test cases (we will call it TDD#3):

Case#	Source	Length	Destination
1	0	0+X	N-X
2	0	0+X	N-X+1
3	0	Y	1
4	1	Y	0
5	Valid	Invalid	valid
6	invalid	Valid	Valid
7	Valid	Valid	Invalid

We'll have to define x and y and in some cases to expand more the definition of invalid data , but it's clear that TDD#3 is much more powerful than the previous two, since it's cover functional aspects of the software.

O.K. the reader will say, you waste our time demonstrating a totally improper use of a test technique which relates to data when the actual focus has to be on the functionality. Would you like that we will add a warning against improper use of a technique on the package?

Yes. We must warn against improper use. when the focus wanders from the cognitive research and analysis to the BKM area, it is likely that we and our reviewers will be less criticizing. If in our example we were starting from the simple abstraction on the drawing board, and only then we will look inside the "techniques basket", picking the ones that we think most adequate for our proposal, we will also give it the proper weight, watching from the technique blindness.
In our example of the last Test design, we might think that few boundary values are missing and that we should relate to both invalid partitions in the invalid input cases, but we will understand that these are only second priority items. since the whole BVA technique is blind for relation betwen the variables, starting with the technique will lead our mind in the less relevant direction.

Changing the nature of this blog

2009-06-30T12:10:00.000+03:00

I have changed my mind!

I don't see any point to dedicate the blog only for reviews. Allthough I am not going to refrain from reviewing other's work at all. So I have changed the title.

I would like to answer Pradeep Soundararajan call ,and go on blogging my own ideas too. I thought about that for a long time, but his last post gave me the push.

Hope to be able and do that soon.

Online test magazines

2009-01-12T09:24:00.000+02:00

The first magazine I would like to link to is Testing Experience:
http://www.testingexperience.com/

This magazine already has it's 4th issue, and seems to collect articles, which expands the knowledge in many areas. So far, I found in every issue 1 or 2 articles which I found interesting, and expanding my knowledge in new domains of of our profession. The magazine also, has it's news section, and the last one, just announced breaking news on....a new quality testing certification... Hey! is this the new gospel of testing? I don't want to start here a discussion about certifications, but this just seems to me as too much. So for the sake of balance, ill point you to a video from STAREAST 2008, which address this subject , and probably will make you smile:
The Angel and Devil of Software Testing http://uk.youtube.com/watch?v=oHtydNQU1f8

The other magazine that I want to review is Quality Matters:
http://www.quality-matters.org/

This magazine has published already it's 1st issue, and seems to concentrate on the foundations of testing. The feature subject of this issue was risk based testing by Rex Black, which is one of the recognized test experts in the industry, allthough too much related to the factory school. The article addressed the foundations of RBT, and could be a good summary of foundations of this approach. So was another article about team development.

Lessons Learned in Software Testing (book)

2009-01-06T09:05:00.000+02:00

Lessons Learned in Software Testing A context-Driven approach, by Cem Kaner, James Bach, Bret Pettichord.

My testlink rating: 5 stars

http://www.amazon.com/Lessons-Learned-Software-Testing-Kaner/dp/0471081124

I thought that it would be appropriate to start this blog with a recommendation on the most valuable( in my opinion), book in the field. This book sits on my cubic shelf, and I use my free time to have one more glance in one more lesson. the book is great for any level, from the most beginners to experts and managers. The thing that I like at most in this book is that it's divided into short lessons, cutting off the accompanying bullshit from the valuable information.

Additional related link - James Bach homepage http://www.satisfice.com/