I came to wonder about oracles in testing. Or more likely got bewildered. You see, I read Cem Kaner's article about this subject and was totally lost when I hit the paragraph "Oracles are Heuristics".
Oracles and heuristics started to blur! I started this post as an attempt to cure my bewilderment via deeper analysis about the article and the thinking behind it, but then shifted to my focus on something much more interesting, namely passing judgment on people.
It's a dangerous path to take, but I took it. Let's see how it went...
Let's start by defining heuristics. I consider them to be specific means to try to solve a problem and learn, rules of thumb, educated guesses, apply common sense or even find bugs, if you will. Oh look, Wikipedia thinks the same! :) But by specific I don't mean algorithms. That's why I underlined "to try". I have a massive library of heuristics. A mindmap of almost 2000 nodes, each node meaning a specific try, an idea to find bugs. It's based on James Bach's HTSM master and updated with everything I got. A lot of it is tied to NDA tied material for example purposes, so I cannot unfortunately share it. But it's basically quite generic, nearly everything can be applied everywhere. And I do just that by selecting the ideas that fit the occasion and context. Adaptation is the key.
Ok, then the oracles. Quotes from Kaner's article:
"A software testing oracle is a tool that helps you decide whether the program passed your test."
"Imagine running a test. The program misbehaves. The tester notices the behavior and recognizes that something is wrong. What is it that makes the tester decide this is wrong behavior?"
Back to Kaner's article:
"(Doug) Hoffman argued that no oracle can fully specify the postcondition state of the system under test and therefore no oracle is complete. Given that an oracle is incomplete, you might use the oracle and incorrectly conclude that the program failed the test when it didn’t or passed the test when it didn’t. Either way, reliance on an oracle can lead you to the wrong conclusion.
A decision rule that is useful but not always correct is called a heuristic."This is where I got lost. And I was lost in a way that for example an ISTQB Foundation level tester might be; I focused on terms and explanations more than the thinking behind them. I did the rookie mistake; I didn't test the rule itself and pass judgment on it.
I wouldn't mix oracles and heuristics together. For me oracles are the flavour of heuristics. Sometimes doing something based on a heuristic might reveal an oracle. Actually that happens quite often. Even more often oracles reveal the need for certain heurictics. But even though they live in harmony, I wouldn't mix them together. Examining oracles and their dynamics is a completely different ball game.
Oracle is, heuristic does.
Oracle helps you to pass judgment.
Passing judgment on... people?
I'm getting nowhere by contemplating the differences of oracles and heuristics so I leave it there. The passing judgment part instead sounds intriguing. As testers, more specificly sapient testers, we pass judgment on software we test. Either it works or it doesn't. Not so easy always, so we seek more information to ease this. One crucial source of this information is people around us.
What about passing judgment on them?
Software is all about people. It's about people doing it, setting expectations to it, using it and depending on it. And people are imperfect. "Flaws" in people's behaviour are all that matters when it comes to hunting bugs. People are the root cause for everything in software testing. Good and bad.
So why not pass judgment on them?
I'm walking a fine line here, so let's be careful...
Few lines before I wrote about implicit requirements/expectations/want. Let's expand that concept to oracles. Indirect, vague and potential means to pass judgment on something. When it comes to software, our goal is often to come up with explicit oracles that leave no room for interpretation. Formal requirements management aims for just this. This is however an unreasonably ambitious goal. That's why we've invented the concept of implicit oracles, which are basically everything that hasn't been or cannot be quantified. Where explicit oracles would be the bricks, implicit ones would be the mortar.
Note: When an implicit oracle is quantified, it becomes explicit. Add this to your process. ;)
Let me elaborate.
The story of imperfect people
Now, have you ever been tired at work? Have you ever been under the weather? Have you had so much to drink on the weekend that you forgot to wear shoes to work? Well, so have the developers, architects, project managers and pretty much everyone responsible of the product/service which's quality you're questioning. Even at full health people aren't perfect. Nothing ever is. The same applies on environments, prevalent mood, weather, alignment of planets and whatnot. And all that affects on quality.
Can we recognize patterns from this mess?
Can we come up with means to pass judgment on people?
Let's pull the breaks here. You should never pass judgment on people! You're never in a position to do so. However people's actions can be judged. The good. The bad. And of course the ugly ones, as you see next.
Let's look at toilet brushes and people not using them. That's a very tangible example. So very many times I've been disgusted by someone's "unfinished business" at the toilet. But when connected to a developer it becomes an interesting idea; Does this person leave other things unfinished too? Very much so! You wouldn't imagine how often these kind of signals can be identified as the original root cause for problems.
Attitude problems, lacking competence, poor guidance, etc. have more prominence in this matter, but I'm more fashinated in subtle signals like developer not using toilet brush. Now that's a revolting thing to begin with, leave places like that after you, but it acts as quite solid oracle; This guy is sloppy, which can cause a number of problems. Ok, that sloppiness can manifest in number of ways, but The Toilet Brush
I'm more alert.
Of course there are many other signals that can raise a red flag, but this and it's cosmic connection to the effects lept to mind. I'm sorry for such a sh**ty example. :D
So if you're not using a toilet brush and I'm next, know that your software is next under the spotlight. ;)
Disclaimer: Be very careful when walking this path. What if you're wrong? What if you're interpreting the subtle signals wrongly? Two tips:
- Start with the more prominent signals. Be sure.
- Notice a pattern before making a judgment!
Those apply quite nicely on software too. If you manage to do this with people, doing it to software is cakewalk. I'd however advice you to start vice versa. So, third point:
- Software first, then people.
- You're not perfect either. ;)
Ok, I've written a heap of text that spans for several days, that has been edited for several times and might miss it's point badly. It's slightly insane and very dangerously interpretable so I'm contemplating if I shoud even publish it. If you see it, you know I've made a decision... ;)
Quote time! One of my favourite comic characters hits the big screen on 21th of September, so let's give room for the king of badasses. He does pass judgment on people themselves, but life's different in post-apocalyptic world... ;)
"It's judgment time!" -Judge DreddYours truly,
Sami "Judge Sir Blom" Söderblom