Wednesday, May 22, 2013

BBST Foundations 2.0 recap


So, I hopped to the BBST bandwagon and attended Foundations 2.0 certification program. The four week course was an eye-opening trip I will never forget. And now, few days after finalizing the course, I'd like to reflect on what happened. With the intent of not revealing too much to those who are about to take the course, of course. ;)

The beginning

My journey started 21st of April 2013. We, the group of 20 to 30 people from all over the world, were assigned a bunch of tasks, the first one being that we should meet and greet each other. The course provided discussion forums for this, but some of us expanded this to personal emails, IMs and even Skype calls. We told our stories, chatted about things important to us and got to know each other beyond the professional surface.

Testing, as any field of profession, is a community. From the very beginning of this course it came very clear to me that BBST values this fact highly. Those who have completed the course are not only wiser individuals, but an engaged group of people who face challenges together.

A nice way to start a course.

The pace is picks up

The cosy start came to a swift end when the first deadline was upon us. The course had two major deadlines per week. Each of them required a number of tasks done, ranging from studying lecture material (videos made by Cem Kaner, articles, whitepapers, etc.) to individual and group tasks.

Recess @ BBST course
The pace was fast. The deadlines required more and more each week and at times I wasn't sure I was going to make it. The course required 10-15 hours of personal time each week, which can be a challenge for many because of work, family and in general life outside this course. For me that often meant shortening my nightly sleep. And that's something you should never do. But the course taught me that testers live and breathe tradeoffs. So, for these few weeks my tradeoff was to live like a zombie. :)

Lessons learned

But the course wasn't all about hugs, kisses and blinding hurry. It was filled with knowledge that many would consider advanced. But it was actually something every tester should know.

Lectures were divided into six increments:

  • Overview and Basic Definitions. The lecture explained the nature of testing, quality, bugs, etc. and defined concepts like black box testing, white box testing, functional/non-functional testing, etc. etc. I especially liked how it elevated the importance of context and stakeholders, the purpose of software and how it expanded my knowledge about the basic concepts of testing without forcing absolute truths or best practices.
  • Test strategy. The lecture introduced an approach to test strategy. Because of this lecture I'm now more structured in my testing endeavours. Complemented with my head and heart, Elisabeth Hendickson's book Explore It! and James Bach's Heuristic Test Strategy Model these teaching give me a solid ground for building test strategies, plans and resourcing in pretty much any environment.
  • Oracles. The course in general is about heuristics, but this lecture concentrated on oracles. An oracle, according to the lecture, is a heuristic principle or mechanism by which you recognize a potential problem. We were taught to apply cool things such as Hoffman's SUT model, signal detection model and best of all, consistency heuristics. Michael Bolton's article Testing Without a Map is a wonderful entry point to the subject of consistency heuristics. But if you want to get to the thick of it, dive into the production of Douglas Hoffman. This lecture vastly improved my ability to make judgments and build models on which to base all testing.
  • Programming fundamentals and coverage. Decimal and binary arithmetics, overflows, rounding errors, floating points, significant digits and precision, bits, words, intergers, etc. etc. I haven't been doing implementation level testing for years and all this was quite a lot to handle. It's still unclear to me why black box software testing course has so prominent portion of unannounced white box stuff in it. I learned something I MIGHT need somewhere in the future (but haven't for years), and I had to use as much time to learn it as I used for all the other lectures combined. And I had no choice over it. That bums me out a bit.
  • The Impossibility of Complete Testing. I've understood the core of this lecture pretty much always as testing indeed targets an infinite and constantly changing space, but now I got some solid examples and arguments. The lecture refined my skills in handling variations and combinations, and helped me to put more thinking into E2E testing, which I use a lot in my daily work. Plus it reminded me of the "ancient" wisdom of Glenford J. Myers.
Red alert!!
  • Measurement. The lecture addressed one of the hottest topics in the field of testing and quality management. I have always had great dependency on metrics and measurements, but day after day I've grown more and more suspicious over them. This lecture pinpointed exactly the things I should be suspicious about and the fallacies of some of the most used measurements such as The Weibull Reliability Model. I learned about the precision, scope, scale and errors of measurements, their characteristics and the purpose of them. But the most fascinating bit was surrogate measures. Wikipedia has an example of them in clinical trial context. The lecture gave a good example of them in software testing context; We don't know how to measure tester productivity, so let's count bugs. Bugs become surrogate for productivity, because we assume that they must be correlated with productivity. And they are also easy to count. The example is very powerful and it ended the lecture portion of the course with a bang.

Together as one

The lectures, articles, whitepapers, etc. were a great set of academic contect in the course, but for me the actual meat came from the assignments. They were individual or group tasks where you were given a challenge. In some you had to define test strategies in different contexts, in some you had to form the context and approach testing from different perspective. The group assignments required consensus from the group, so one of the challenges was to fit different opinions into one logical output.

All of the assignments were highly demanding and they drove people to accept that failures are inevitable. I came to think of old samurai wisdom; The best way to survive a battle is to accept that you're already dead... :) I learned a great deal of humility and teamwork during these assignments. The course gave us the facilities to spar our ideas with people from very different backgrounds, cultures and geographical locations. Testing is also about advocating your ideas and observations, and this was beautifully implemented into these assignments. The course constantly made us to make our point and to require that from each other too. Every claim was tested, every idea questioned. Even the final exam followed this pattern; We were put to answer a set of questions AND review our co-students' answers. It was not only about our ability to understand what was taught, but to make fair judgment on others' understanding too. This is something quite unique.

I'm now BBST Foundations 2.0 certified tester. Yay! :) It's however not because I did a good individual performance, but because I did it with the help of wonderful people. Love you guys!


If you've read the whole thing, you probably got the idea why. :) Even though it was a demanding trip, it was also a rewarding trip. I learned a heap of new, refined a bunch of old and eventually came out as a better tester. And best of all, I'm already using the skills I learned in my daily work. And I'm teaching them to others too!

Yay! :)

The world of testing is a bit clearer to me because of this course. I'm now more prepared to face the challenges it throws at me. But it's not all challenges and battle. It's also about endless possibilities, fun and people whom I cannot wait to meet again.

I'll se you at BBST Bug Advocacy course! :)

Kind regards,



  1. Great post, Sami! And welcome to the BBST family! Congratulations!


    1. Thank you Teri! It's good to be a part of this family! :)

  2. Congratulations on successfully completing BBST Foundations. I wish you the best of luck in BBST Bug Advocacy. We are always delighted to see thoughtful reviews of the courses posted on blogs. Thanks for the time and effort it took to write this review. It's a terrific summary and I've added it as a link on the Testimonials page at

    1. Thank you Rebecca! Very kind of you to add this to

  3. Good post, interesting comments!

    Your feedback is useful as I make ongoing revisions to the BBST courses. Thank you for it.

    Regarding Oracles -- I added some notes on oracles to my version of the course: I'm not sure whether these have made it to the AST version yet, but some of my students have found them helpful.

    Sorry that you didn't like the material how programs handle data. Adding this was one of the major changes from Foundations 1.0 and from my earlier versions of the course. When I used to manage testers, staff of mine would sometimes make fools of themselves by demonstrating ignorance of basic issues in computation (especially the accuracy of calculations or matters of coverage and testability). It seemed to me that for someone who (black box) tests software that calculates with numbers, not understanding floating point math is a dangerous thing.

    I found similar problems when I started teaching (1993-2010) versions of BBST. It was such a problem that I abandoned using OpenOffice Calc in the Bug Advocacy sections of the course. Some students couldn't understand the issues raised in the bug reports (for example, they didn't know how to assess whether the result of a calculation was within tolerance limits or not) and made comments that reflected the lack of understanding. Other students filed new bug reports that were inappropriate. The responses from OpenOffice team members were sometimes harsh and disheartening. In addition, in the test design sections of the course (some of which are in BBST Test Design, some will be added to later courses), some of the black box techniques befuddled students who didn't understand enough about how software works and how data are stored and manipulated.

    I could see these problems particularly clearly once I started (in 2000) giving students assignments and exams that I could review in detail.

    When I redesigned Foundations, my goal was to add the minimum amount of technical material needed to plug the gaps I was seeing reflected over and over in exams (on black box, not glass box, test design) and assignments (on design and in bug reporting).

    Just as signal detection theory and much of the measurement theory that I presented in class come from disciplines other than black box testing but shed light on black box testing, the same intent applies to the programming material--to aid black box testers to understand what they do and the contexts that they do it in.

    No selection of material can be perfect for everyone, and I apologize that this selection didn't work out as well for you as you had hoped. I am glad that you found the overall experience worthwhile.

    -- cem

    1. Thank you for your comments, Cem!

      There's no need to apologize for anything. You've done a heroic service to the entire testing industry with this program. Even though I didn't benefit from the white box portion of this course that much, many did. Especially more technically oriented testers really dug deep into this subject and had discussions that even I gained a lot from. I agree that even I should know these things even though their practical use is rarity for me.

      But I'm indeed just one student among many. I often say that there's as many approaches on testing as there are testers, and to serve every need is quite impossible. So it would be unreasonable for anyone to except perfection from any course.

      My critique was however subjected to the fact that I didn't know how much effort the white box portion of this course required. If I would have known about it, I could've allocated my time more effectively. Perhaps more refined/detailed course description, active instructorship or something could help in this.

      Either way I had a blast, learned a lot and would've taken the course no matter how big the white box portion of it was. :)


  4. Thanks Sami for your write up.

    Cem's response to you reminded me of his legendarily awesome response to a stackexchange question.

    Talk about going above and beyond to improve the field of software testing!

    Justin Hunter

    1. Hi Justin,

      I'm currently working with Lean enthusiasts and this response fits beautifully into our discussion. Thanks!

      Seems that every time I even whisper the word "waste", these guys go apes**t! :D


  5. Thank you, I really appreciate this!

  6. It’s very helpful for us, thank you so much for sharing such an amazing article. Visit Ogen Infosystem for top Website Designing and PPC Services in Delhi at an affordable price.
    PPC Company in Delhi