Friday, April 1, 2011

Should the testers be responsible for pinponting the cause of Defects ?

A tester works fabulously to find an important bug in an application and raises a bug report with lot of excitement and possibly all the details that he could include. The bug is 100% reproducible. After some days, the bug report gets assigned back to him stating that the Developer needs more information. Upon further analysis of the request, the tester figures out that Developer is actually requesting for assistance to help him "pinpoint" the problem in the code. Of course, the Developer is not asking the tester to go through the code but asking more intricate details that are beyond the scope of a bug report (as the bug is 100% reproducible). The tester is in confused state! He thinks that he provided all the information that was there to be provided and the bug is 100% reproducible, then why on earth would a developer need more Information. Let’s look at a perspective about this problem-

Is pinpointing problem a part of test group's job?
In my view, in an ideal world, in couple of words- It should be. Let me explain my perspective here.
Project teams develop Software to serve the customers. The test teams primarily exist to provide Specialized Services to the overall projects. The extent of Service include “what” and “how” of it and primarily depend upon the overall Testing mission we choose. Generally, one of the common views is to consider testing only as “Detection” and “Reporting” of issues. That certainly leaves out pinpointing activity. To simply put, pinpointing a problem is a activity that helps a developer figure out the buggy part of code.
It’s no wrong in treating testing more alongside the “Detection” and “Reporting” areas but when we talk about the notion of Service, one of the things that get associated with it is Value generation. In my view- In any Service related profession, more than anything else, the overall Value should always be upwardly mobile. Value from the testing group as such is perceived differently by different people who we work with and provide our Services.
Testing all that’s possible, Providing an Accurate flawless defect report, Providing accurate bug related information (metrics) that can help Management make crucial product decision, Helping support team giving crucial information on a customer problem etc. are some Value generating activities that goes beyond the idea of Detecting and Reporting. Of course, Pinpointing problems is another activity that helps add Value as a developer perceives it from the Test group. I know in some cases test teams even helped in fixing the bugs, so essentially it boils down to what “mission” do we commit to.
As Gerald Weinberg also states in his book Perfect Software: And Other Illusions about Testing
Testing (can be) for Discovery, pinpointing, Locating, determining significance, repairing, troubleshooting and testing to learn.

As I mentioned in the beginning, the above description represents and close to Ideal world situation.

Is it really possible to have the clear distinction between pinpointing and testing?
I think No, its not possible to create that distinction especially when we consider the product from a black-box perspective. If we do White-box testing, it is possible to have some sort of clear distinction. I think more or less every good bug report has some level of Troubleshooting done. But “some level of troubleshooting” is often not as accurate as “pinpointing”. Pinpointing just helps the developer to reach the faulty lines of code, identify the problem and fix it. Apparently, in a product with millions of lines of code reaching that faulty line of code is a very tedious job (sometimes a mystery!). If a tester helps a developer do that, then we have a delighted customer (i.e. if at all we treat him/her as a customer. In normal circumstances, we should). From test perspective, it is again the amount of cost we are willing to spend to reach this level of delight. Again, it leads to the eternal question- “what is our mission”.
In short, it is not possible to have that clear distinction between pinpointing and testing in our kind of test setup.

Does the time-limit heuristic usually work always ?
Gerald Weinberg also talks about Time-limit heuristic.
A heuristic that helps untangle who does what for how long states the concept simply:
Nobody on a project should carelessly waste the time of anyone on the project

According to this, the test team member sets a time limit and communicate to the Development team on how much time can they spend on the Development specific investigation. Beyond this time, the onus is on the Development team to figure out.
Since this is a heuristic and not a rule, its quite hard to define the time-limit. Only way it can work is if we make it as a rule i.e. Test team would not spend more than 1 man-day or whatever is the limit and track it accordingly. Doing so (without prior agreement) , will cause some grim repercussions including the relations with Development team.
Having said this, testers don’t have endless time at their disposal. So, till the time there are a formal guidelines around this, it's good to make the Development team (requesting the pinpointing Service) aware of the extent we can help without causing the Testing schedule in Jeopardy.

If the bug is reproducible, to what extent should the test team help the dev pinpoint the bug?
The greater degree of conflict (regarding pinpointing) between Dev and Test arises when the bug is reproducible 100% of time. Ideally, the Dev should Investigate, debug and figure out where the bug is in the code with some help from Test team.
If the bug is non-reproducible then there is even finer distinction between Bug detecting and bug pinpointing. I think Test team should lead the Investigation here whereas for Reproducible bugs the Dev team should lead the investigation.

Closing thoughts:
Considering the above points, I can think of the following points-
• Get in agreement with the Dev team on what level of pinpointing can the test team help with. It should be made clear at the start of project involvement.
• Have a Clear distinction on what Activities include Detection, Troubleshooting and Pinpointing. Possibly, include the same in the Test plan.
• Include Pinpointing activities as a part of Standard Test estimations. It won’t be too high a cost considering the entire span of release but it is worthwhile to consider it as a separate line-item.
• Be aware of impacts of frequent Task-switching for testers (which usually happens when testers switches to Bug investigation away from testing work). Not everyone can do it with an ease. This can be one area specific training can be designed/conducted for testers if there is a need.

Any thoughts ?


Rajesh Kazhankodath said...

This post is a good summarization of the very practical situation encountered by software engineering teams.
Part of the testers credibility lies in his or her ability to get defect fixed. The skills required to getting defects fixed goes much beyond the art of finding defects. The first step in getting defects fixes is to detailed and informative defects reports.
Testers will naturally will extend what ever help required for the developers when its part of their responsibility to get defects resolved rather than just finding the defects. It also helps immensely to keep the defect report updated with details of the activities performed by both developer, testers and every interested party.
Testers and developers need to have the shared objective of delivering a quality software to the customers and therefore collaborating with each other in resolution of issues is very important.
Another important aspect to consider is that expertise gained on the technical as well as functional aspects of the product when testers are debugging and isolating the issue. On the long run, this expertise will much more value than the time spend by them in debugging the issue.

Anonymous said...

Let me guess ... you're a developer.