Since I started leading my current testing team, I’ve been struggling with the test case base.
The First Sin: Detailed Gen 0 Test Cases
The reason for the failure is the limitation of our mind to perfectly imagine an abstract design. Sometimes even the designers doesn’t have a 100% complete design. While you can plan many things ahead of time, you can also anticipate that you will have gaps in your planning, but not anticipate their exact location.
The Second Sin: Detailed Gen 1 Test Cases
What about tests that successfully made it from Gen 0 to Gen 1 and proved to be correct? What about tests that were designed after the product was introduced and tried? They might not suffer from the first sin, but they will suffer from the second sin. Although these tests were accurate in the assumptions about the product itself, not all of them were the correct ones to run. Moreover, some of the tests that did a great job for Gen 1, finished their duty. Using these tests in regression will not be efficient.
As we progress with the test execution, we learn more about the risks of the product. At the end of the first generation testing we can plan better regression testing for the next generations. Typically we will add a small number of test cases and get rid of a larger amount of tests.
Conclusion: investing in too many detailed test cases during Gen0 and Gen1 is not efficient.
I’ll try to define basic guidelines to deal with this issue:
1) Lower the expectations from Gen0 and Gen 1 test cases – understand the built-in limitation of these test cases: Gen 0 might be inaccurate and Gen 1 will not fit your regression needs.
2) Seek for alternatives when planning Gen 0 and Gen 1 test cases. For example, use checklists instead of steps (See “The Value of Checklists and the Danger of Scripts” by Cem Kaner).
3) Try to thinks of better uses of your time during test planning. For example, invest in automation infrastructure during preparation.
4) Realize that moving from Gen 1 o Gen 2 will require more time in test documentation, and is not just copy-paste from Gen0-1 test cases. In this stage, you can save time by creating less “perfect” test cases of new features in the same product introduced at this time.
5) Consider the possibility that you will come to like your lean Gen 0 and Gen 1 test cases so much, that you won’t want to invest in more details for the regression test case base.
In case you claim that your experience is different and it’s possible to create perfect re-usable test plans in early stages, I can think of the following possibilities:
1) You are a better product and test designer than the ones I work with (please mentor me).
2) You don’t have complex and innovative products like the products that I test.
3) You follow a perfect process that prevents you from falling in such traps (and I would like to hear more about it).
Thank you Issi. You made me rethink about the process of creating a test plan. I will try to implement.
ReplyDeleteErez, It's a pleasure when a near colleague reads my posts. it's a double pleasure when he find them worthy :-)
ReplyDeleteI would love to hear your experience in terms of real changes
Dear Issi,
ReplyDeleteThat's a great post, and when read in relation to what Joel M. and Gojko Adzic posts, might require a major rethink on our way of work.
( http://qablog.practitest.com/2011/07/when-your-job-is-not-to-test
http://gojko.net/2011/03/03/simulating-your-way-out-of-regression-testing )
This goes well with my past claims, of having to rewrite the "Atomic" tests written for stages 0&1 by your vocabulary, and rethink them before we approach the regression stages.
From my reply to Joel's post:
"On the other hand, I do agree that we waste too much time & effort on Regression, which yields relatively low amount of bugs.
From my experience, most of the bugs raised in automatic regression, were raised during the debugging of these test suits, or while reusing them on newly developed units which share much resemblance. (again - you can't call that regression yet).
I have claimed in the past, that we tend to execute most of our regression tests - just since it’s there, without constantly verifying the usefulness of these actions.
More than that, I claim that we tend to use atomic tests which are well written for the initial release of new features, and instead of throwing these away and designing much more useful combined test cases for regression, we drag around with these redundant and useless tests onto regressions of next releases."
>>> One of the problems, is that regression is not as far as people think - it already arrives during middle of 1st major release of a feature, while we are in time pressure and unable to spend time in thinking about what we do, let alone write that down to any level of details.
So where does it place us?
1. Don't waste too much time in Gen 0 - But DO PLAN, review and pin-point important areas - just don't mind the details and phrasing too much.
2. Plan ahead (before and during Gen 0&1), for Regression tests - consider that, mark relevant tests within Gen 0&1 STP/D, and leave time to improve these during Gen 0 & 1.
3. Recheck these at proper time - set a review when reaching 50% of Gen 1.
4. Based on Gojko & Joel - Try to write only useful regression tests, showing necessary coverage to give confidence of these parts - without expending and expecting to find new bugs.
5. Automate accordingly - and mainly set infrastructure.
(Of course, you might use intermediate automations for unit tests, semi-manual tests etc. where it improves productivity)
Thanks Kobi for your comment and sharing these important ideas.
ReplyDelete(You should consider to start your own Testing blog too)
The two related posts you mentioned, set me the trigger to write this post, which illuminates the subject from different angle.
Thanks Issi,
ReplyDeleteI find that Blogs are not interactive enough for me, it's like a lecture which do not invite much feedback.
I prefer to share my ideas in Forums, which invite other people's feedback, which makes the discussion flourish, and new ideas are learned from that.
Also - I do not believe I would have enough good ideas to keep a blog active, a "Group" blog might be more active, thus attract more audience more frequently, reducing the need to track so many sites.
So now I focus on gathering information from different links, and discussing them in a central place (while giving credit to originators, and increasing the traffic to their sites - so it's a win-win I hope :-)).
Kobi