The World Divided

Hi,

So. As I promised, here's some more concrete stuff. Not rock solid, but at least something to hold and hug. I'll write about a subject that has been close to my heart for quite some time; The World Divided. It is of course the war between the people representing the traditional way and the new way of doing testing. Ok, those who do testing in more traditional way might not know the difference, so I'll shine a light to this...

The traditional way of doing testing, as I see it, is usually referred as plan-driven testing. In this kind of approach you have strictly phased development and thus testing, clear rules and processes how everything moves from development to testing and back again. There are usually comprehensive test strategies, plans, test cases and whatnot to make sure that everything proceeds like a train, everything is coverered and taken care of, at least in plan level. Often this kind of approach is applied when there's a lot to control and the fear of losing this control is great. From my point of view, the traditional way of doing testing represents the very basic level of thinking and is often the safe way of starting testing. It can also enslave your mind if you're not careful...

The new way of doing testing, as I see it, is usually referred as more agile testing. Agile might be a bad word here because there are for instance Lean practises that represent the new way of doing things too. I don't know about Lean that much, so let's just skip that and use just Agile, m'kay? There testing is done in close collaboration with development and other stakeholders. Agile Manifesto says it best:

Individuals and interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiation 

Responding to change over following a plan

Noble goals, huh? There's a lot more to it than this and realizing Agile, but unfortunately only few know how to apply it correctly, to follow these four goals. I might fail in following them too, but at least I try. Agile approach is ofter applied when there's not that much to control, not much to lose, and by more open-minded people with less change resistance tendencies. If Agile proves itself, it can be taken more seriously and applied in bigger and more complex projects and environments; Something that the traditional way doesn't have to go through.

Think different!

Actually there's a lot more to both of these two worlds, but now I'm not trying teach these thoroughly, but just to give an idea of the differences between them. You might even picture what kind of people do the traditional way and what the new way. I'm a bit biased in this; I feel that the traditional way is the bare minimum, achievable by anyone. I've seen people with zero experience, zero skills, zero vision pass certifications (which represent the traditional way quite strongly) with flying colors. If our cat could read, he'd pass those. The traditional way of doing is filled with pre-conditioned processes, methods, check-lists and all kinds of gimmicks to make sure that everyone is capable of doing testing, and therefore make individuals replaceable. This machinery makes it really hard to spot real talents... *sigh*

At this point you should read this article about Big Macs vs. The Naked Chef. It's absolutely brilliant!

Ok, I'll ease the slander. I just feel that the new way represents the higher level of thinking. I've seen people doing testing faster, more accurately, more creatively, [insert appraisal] and have generally more fun while applying the new way. But it comes with the responsibility. For instance Agile is often mixed with ad hoc practises i.e. structure and discipline are gone. People are sitting in a meeting room for 15 minutes every morning, and nothing else, just to get that Scrum stamp on CV, or just playing with systems to get that sexy Exploratory Testing Expert reputation. Agile is often abused by those who should stick to what they know (i.e. the traditional monkey way), and thus given a bad name that good people are trying to fix through hard work. For instance the last project I was in suffered greatly from the misuse of Agile practises and unfortunately that was not the first time for me. This creates the beforementioned bias and even deepens the moat between these two worlds.

But there's a hint of hope left in me.

Just enough to come up with a solution of sort.

The Hybrid!

I practise jujutsu and I read this book about Miyamoto Musashi and the school he created, Niten Ichi-ryū i.e. the school of the strategy of two heavens as one. Somehow I started to think about the two worlds of testing. As Musashi used his two swords to complement each other, I started to think about combining the teachings of the traditional way and the new way of testing. And I came up with the hybrid of plan-driven testing combined with exploratory testing...

About Exploratory Testing

Exploratory testing is often considered as an opposite to "scripted testing" (i.e. traditional plan-driven), because there test planning is done during testing, and changes all the time depending on the test results. Tests cases/scenarios/charters/etc. are not planned throughly beforehand and they are not exectuted in certain way. Emphasis is put to understanding the test subject and how this subject should work.

Exploratory testing is often mixed with ad hoc testing, which is the loose and sloppy way of doing things. It is however very refined, scientific thinking done real-time. Although it doesn't get the respect it deserves among so called test professionals, every tester has used it in some part of their career...

The Elements of Exploratory Testing

As I've come to believe, exploratory testing consists of five elements, that shouldn't be considered as phases, because every element can be applied quite freely without the need to tie it into any timeframe.

• Exploring the subject. Exploration and mapping of test subject's purpose ja functions. All source material can be used, even the earlier versions of the very subject under exploration. Aim is to understand how the subject works and how it should work (stated and implied needs of the buyer), is there any pitfalls or even defects present. Experience, test techniques, technical competence, etc. can be harnessed to help in this. Experienced tester can estimate workloads, focus and resource testing, and generally prepare for upcoming battles and tackle possible problems before they even occur.
• Planning the testing. Definition of strategy for how tests will be executed and how the test subject will be exposed to the abuse that is testing. Freeform and malleable workplan of how many and what kind of people is needed, needs for configurations, equipments, work environment, etc. and to support that somekind of plan of what to test at first. After testing the first, based on the results, the second is planned and tested, etc.
• Testing. Doing the actual testing, inspecting and exploring the subject, questioning it, digging out the bugs. As said before, plans are adjusted based on test results. See: Session-based testing.
• Heuristics. Applying the means to test, thinking your way through. Reasoning, problem solving, creativity, analytical thinking... Break the rules if it helps. Be a smart human.
• Results. Exploratory testing, as everything successful, does produce results. Testing stops when it's shown that everything that needs to be tested is tested. This is quite loose definition, but it's good guideline to stick with the requirements. Tester has to prove that set requirements are covered via testing. Even though exploratory testing is very dynamic and intuitive, even fun way of doing testing, it's imperative that it produces results; Clear and effective reports that unambiguously depict the quality level of the subject tested.

You, dear reader, might have more of these, so please do comment!

                 Portait of a tester               

Exploratory Tester

There are tons of qualities that a good exploratory tester has, and I'll write about those in my uncoming post The Qualities of a Good Tester. In short; I try to be a good exploratory tester by writing up ideas beforehand and using them when I'm testing. I also study the field of testing to the limit and beyond, try to develop my mind, and step constantly off from my comfort zone to learn more, and to find bugs of course. But one of the most important qualities to have is to recognize when it's wise to use exploratory testing and when it's not.

Exploratory Testing vs. Scripted Testing

Confrontation between exploratory testing and scripted testing should be avoided, because they can complement each other. To which the emphasis is put has to be recognized from the nature of the project, people in it and it's test phases.

If there's problems with resourcing, time or quality in any level; if requirements, specifications or planning aren't quite what they should be and leave too much room for interpretation, exploratory testing should be take into use at least in some level. Often exploratory testing is used most successfully in acceptance phase where business evaluates if the subject is good enough to be released to pilot or even production, but I've seen it been used quite well in other phases too.

All in all, only perfect plans and perfect realization of them ensures that there's no room for exploratory testing.

Basic Company X Applying Exploratory Testing

Let's take Basic Company X under the magnifying lens. This company can be anything; It can operate in the field of video surveillance, advertising, insurance, banking, telecom, gaming, retail sales, logistics... you name it. It's testing is strongly emphasises on the traditional way of doing, on scripted testing. So it produces a ton of requirements, test plans, test cases, etc. material and spends quite a lot of time producing and managing that. Majority of companies operate like this so it's quite obvious that Basic Company X does so too. It might work to a certain point, but there are certain risks that need to be taken under consideration:

• Human errors happen. Whether it's setting requirements, writing plans, executing tests or even crossing the road, we all make mistakes. The nature of testing is that we seek what is caused by mistakes. Often problems are found in later phases of project, because not until then the means of professional testing are applied. These findings create changes to requirements, plans and execution. Often problems are found from the areas no planning covers, or worse, are not found at all, because this not-so-good planning doesn't lead into problems, but is however followed blindly.
• As said before, production and management of requirements, planning, test cases, etc. takes time, creates costs and thus lowers ROI that is gotten from testing.
• Strictly aligned, to atomic levels planned test cases enslave the minds of testers. Tester that follows a plan without questioning it cannot find anything and is therefore useless.
• Over time, detailed test planning starts to affect negatively to those testers who are capable of doing exploratory testing. They lose their edge, cannot find anything and become therefore useless. Ok, they are not totally lost, but they need to be taken out from this kind of concept quickly before something valuable is taken from them. This kind of "robot activity" can be treated with test automation, but it's still not a cure.

The safest way for Basic Company X to apply exploratory testing is to follow set requirements, but to be flexible when it comes to following and updating plans. This is what I've done to sell exploratory testing to those who feel reserved about it:

1. You plan your test cases/scenarios/charters/etc. by your best knowledge and practise.
2. Start your exploratory testing sessions, but without interfering with the plans you made in part 1. Remember to write down what you did!
3. You take your plans made in part 1 and reflect the results from part 2 to these plans. You definitely did something you planned on part 1, and with the rest you update the plan.
4. Finally you test the cases/scenarios/chartes/etc. you planned on part 1, and update them if needed.

This way you maintain the discipline of scripted testing, but in the same time gives the freedom to test more creatively without enslaving the minds of testers. Test statistics scale, but are constantly updated and produce valuable information to stakeholders to build their decisions on. Raw resource benefit this doesn't give unless something is taken away from test planning, but the intellectual and motivational effect on testers is beyond measure.

Pros of Exploratory Testing

Here's some pros I've noticed about exploratory testing:

• More multidimensional, creative, intuitive, thought provoking, [insert appraisal here] way of approaching the test subject and problems in it.
• Encourages testers to think and find bugs in ways that would never occur when doing things more traditional way.
• Unleashes all the potential of a professional tester for the benefit of testing, quality and buyer.
• Lowers the threshold to begin testing and in best case makes it enjoyable, even fun.
• Ready to be used immediately, because planning is done when testing.
• Important, high risk problems are found and delivered to be fixed immediately.
• Exploratory tests are not planned on atomic level so they can be used to produce different results each time by e.g. varying test data. Results are however reported by that mandate that test has to be able to repeat as the same.
• Exploratory testing can be used to verify that testing done in previous phases has found the most important bugs.
• If requirements or plans are not quite what they should be, exploratory testing fits like glove.

There might be more, so please comment!

Cons of Exploratory Testing

On the other hand there are also some cons:

• Beginner in testing might feel it awkward to try exploratory testing. Basics first, but beware of losing your mind into the traditional nonsense!
• Basic Company X produces high quality products with high demands, so exploratory testing alone cannot be used due to it's dynamic nature. However it can be used in collaboration with scripted testing, as mentioned before.
• Test cases/scenarios/charters/etc. cannot be reviewed by the buyer before executing them so the quality of those is unknown.
• Creating reports and gathering statistics to a presentable form i.e. telling what works and what does not might be difficult during testing. Verbal feedback and promises are often empty and should not be considered over measurable data.

There are definitely more of these, so please comment!

Conclusion

It' important to know that exploratory testing doesn't solve all your problems. It's not cure for cancer. It's just a rather simple way of doing effective testing. It may not create a sense of structure and stability as scripted testing does, but no one can deny it's effect on testers; At it's best it can help testers to become better at their profession and start to think what they are doing.

So, quite the marathon! :D It's nice to spread wings and write for the joy of it, and I did just that. Hopefully you got something from this, dead reader.

Final words are from the article about exploratory testing that inspired me to write all this:

"The basic rule is this: exploratory testing is called for any time the next test you should perform is not obvious, or when you want to go beyond the obvious. In my experience, that's most of the time." -James Bach

Yours truly,

Sami Söderblom