Sunday, February 7, 2010

Motivating (or relentlessly Pushing ?) Testers towards higher number of bugs. Does it always work ?

Numeric defect related goals. Have you experienced the syndrome ?
Have you ever heard a team owner (read Leader, Manager or whatever) of a Testing team say any of the below-
- My study of Defect Prediction model suggests that this project should have 400 bugs. Why have we found only 50% so far ?
- You have logged less number of bugs in this cycle. Please pull up your socks.
- Your performance is not upto the mark. Please log more bugs.
- Your goal is to log 50 bugs in this cycle.
- In this quarter, you need to improve your bug logging rate.

If you haven’t heard any of the above in your Testing experience, I would consider you lucky, you have been in a good company. Given the professional situation we stay in, i am convinced that many of the test engineers might had to go through the situation where in they were assessed or recognized solely based on the number of bugs they report, no matter what that means.

Is it worthwhile having Numeric defect related goals ?
It usually makes me wonder about the benefits of having the numeric goals bestowed upon the Test engineers. I really fail to find many.
One may argue that having a numeric goal as a reference always keeps the test engineers focused on the goals that they need to achieve. This may be fair for some but this explanation does not capture the overall scheme of things. Consider the below quote by Ichek Adizes that i recently read-

Managing only for profit is like playing tennis with your eye on scoreboard and not on the ball.

To me this quote clarifies a lot of things. With due credit to the author of the quote, i would take the liberty of tweaking this quote a bit for our profession. Read this-

Managing only for number of bugs is like testing the software with your focus on a mere number to achieve than for the love of profession.

Doesn’t the above quote (or requote i say) brings out the essence of the ill effects a senseless numeric goals has on the most important assets of a testing group- The people. The people who are passionate about Software testing would resonate that what interests them towards this craft is the process of learning something new about the Software, having or working to build a unique perspective to assess the software, to solve the problems of complex nature, to apply new techniques, to learn and relearn and many many more things. And the end result of application of this passion, interest, skill is the bugs that we see testers find.
Having focus only on finding a certain number of bugs seem more like robbing a Test engineer of all the greatest perks of work that result in finding the good bugs.

The need for Positive motivation to find bugs:
Before I sound like going unidirectional, I do recognize there is a need of certain motivation that can possibly help find good defects. And certain people do apply the thought of keeping numeric goals to improve defect rate and probably may achieve certain success but this certainly leads to defocusing of Test engineer's energies to a mere number leaving him/her not enjoying the overall process of testing, which is certainly not good for the overall sake of the profession.

I do believe that in addition to educating Test engineers on the right knowledge and skills, as a Test group owner one needs to have effective motivational strategies in place as well. How-so-ever tempting the business of numeric goals may sound, somehow it doesn’t seem to convince me fully. I would certainly worry if any project in question doesn’t have any bugs found but then such a situation is ideal for further analysis and to be figured out the root cause of such an issue rather than just setting a number goal. This may work sometimes (may be if you are dealing with staff who isn’t passionate enough about Software testing and treats it as just as a Stop gap arrangement) but not in majority of situations.

Try using Placebos for silent motivation:
In his book, Stop the Excuses , Dr. Wayne W. Dyer says-

That mind controls the body is hardly up for dispute. You have probably heard of documented studies where sugar pills given to control group believing that they're a remedy for, say, arthritis, turn out to be as effective as the drug being administered, for arthritis. This placebo effect apparently occurs due to the belief in the effectiveness of the pill.

Placebos are quite widely used in the field of medicine. It’s a kind of make believe treatment in which patient believes that he is being treated upon with medicine for some ailment but in reality he is just being sugar pills. This makes his mind believe that he is being treated rightly and to some surprise the medical field has found considerable success with such a Pseudo treatment.

Placebos and Software testing- Are they related somehow ? Some might think this reading the above. But i feel there is an inherent connection there. Consider a situation that you are dealing with a complacent team who believes that they have tested all that could be possibly tested in a product and there are "no" bugs in the product (In my experience, I find teams feeling confident, may be over confident to make such a tall claim, which is improbable). In these situations, one could either believe what the team says or may be "push" them to find more bugs by setting numeric goals. But I do feel Placebos do work quite well there. Just have the team swallow a Placebo that "Development team feels that there is some part of code which is vulnerable to bugs, many bugs." I call this Placebo because as this is no less that a sugar pill that helps Testing team deal with Complacency and act as a certain source of motivation. Complacent test engineers, no matter how much skilled they are- tend to let go of opportunities to find good bugs. You might even be surprised to see the number of test ideas that starts flowing out of the minds that weren’t even considered before. All this just by chewing (not literally!) an artificial Placebo.

The overall point I was trying make is that there are good ways, better ways than mindlessly directing numeric defect related goals.

Do you tend to agree or disagree ? I would certainly love to hear!

3 comments:

Vittal said...

Hi Anuj,

First, Thanks for the wonderful article you have shared and explaining The PlaceBo concept.


I had a different thoughts on this.
Should number of bugs be the only criteria or should betterment of the product be the focus.

As a tester my goal is to make a Product /module stable, in doing so, i should be doing what ever it takes, which means, i should do loads of testing keeping various aspects in mind, like functional part, user scenarios / user stories, performance ...... and the list continues.

Invariably all these will lead to finding bugs and more bugs.

As per me, setting a numerical goal for the team will definitely not be a motivating factor.

Instead of focusing on the number of bugs/count of bugs, if we start focusing on testing itself, like introducing new ways of testing approaches like Exploratory testing, session based testing, assembly line, soak .... rapid testing, extreme testing and the list goes on.. , we will be able to achieve both the goal of keeping the team motivated with new learning, finding more and more bugs and also helps improving the product quality.

I am definitely not claiming creative methods help find 100% of the bugs nor i am against the traditional approach. My view is, when be do a careful mix of both traditional approach of testing and creative testing we cover a large portion of testing spectrum.

All i want to say is, team owners should equip tester with right skill set and train towards the right approach towards testing.

Improving the product is definetly a motivation to test more.

As i said, these are just my thoughts.


Regards,
Vittal

Anuj Magazine said...

Hi Vittal,
Thanks for visiting and for your comments. You have brought forward some nice thoughts.

As a tester my goal is to make a Product /module stable,
Slightly differ here, as a tester my goal is to actually to provide information to the stakeholders thaat eventually help to make quality related decisions. In doing so, I ensure that bugs are logged timely and provide adequate information (Defect Advocacy) to be able to appropriately relay the defect information and hence get them fixed.

i should be doing whatever it takes, which means, i should do loads of testing keeping various aspects in mind, like functional part, user scenarios / user stories, performance ...... and the list continues. Invariably all these will lead to finding bugs and more bugs.

I agree but at the same time depending upon the role one plays in organizations, it restricts us from thinking beyond what we are responsible for. E.g. Especially in bigger organizations, If my team is called Functional testing group i might be expected to think and do only Functional testing thought not on other aspects of the product testing.
My point is sometimes such classification of groups restricts one thinking. But one should be smart enough not to fall in this trap.
As you seemed to indicate, Free thinking is the key here apart from the necessary skills to do any specialized form of work.

Instead of focusing on the number of bugs/count of bugs, if we start focusing on testing itself, like introducing new ways of testing approaches like Exploratory testing, session based testing, assembly line, soak .... rapid testing, extreme testing and the list goes on.. , we will be able to achieve both the goal of keeping the team motivated with new learning, finding more and more bugs and also helps improving the product quality.

I full endorse your views here.

vittal said...

Hi Anuj,

Thanks for taking time to reply to my comments.

One the 1st part,
I agree and full understand your point now, as a Tester our job is to give quality related information to our product stake holder.

On the second part,
Yes, we are restricted within the roles defined, but test group owners should encourage free thinking.

Thanks,
Vittal