Sunday, June 8, 2008

Designing effective Test status reports

"You are the headlights of the project"- Sounds something familiar? Yes, it is the very first lesson of the landmark Software Testing book- "Lessons learnt in Software Testing". The intent of this lesson is simple- Testing is done to find information . This is infact the essence of testing group. The project stakeholders always rely on the results of the testing effort to get a feel of how the project is coming along. And it is this information that helps executives take important decisions about releasing and positioning the product under test in the market. That is probably one of the important reasons why organizations invest in an independent testing group.
There is no doubt that finding information is one of the premier role of the Testing group in an organization but what is equally important is how effectively the data is actually presented. I was reading an article on Metrics by Mr. L.R.V Ramanna and the below mentioned quote took my attention-

Test metrics are like corporate balance sheets. What they show is important but what they hide is vital.

So Test status, depending upon the way it’s presented- can help take Critical decisions with ease or rather than answering questions, raise more questions. Consider, a project is 2 day away from the release and the testing status report tells only about the Open defects and the defect trends and misses vital information such as Test coverage etc. may cause decisions to be made without enough considerations to the facts unless somebody raises the hidden questions

Here i am sharing some tips based on my experience on how the status reports can be made more effective.

Identify the audience of status report:
Knowing the audience who will read the report is an important exercise. More often, i have seen that the Test status reports are sent to distribution lists containing large number of people without a thought to who the recipients are. Having knowledge does not mean to know the people personally, which may not always be possible if you are working in an organization distributed world wide but it means to understand their role in the project. This always helps to anticipate the expectations people have from the status report.
It is always preferable to list the role of people receiving the status report which may be as diverse as the following-
- Product Manager
- Project Manager
- Development Manager
- Development Director
- Test Team
- Development team
- Vice President etc.

If the status reports are sent without knowing who is receiving it, then it often leads to majority of recipients ignoring the report upon receiving it and you take the blame of unnecessarily spamming their inboxes.

Anticipate what questions the recipients would have:
This step is the basis of Test status reports.
The key purpose of Test project status reports is that it answers the wide variety of status related questions that the project stakeholders might have. An effective status report would always answer the diverse questions clearly. Depending upon the phase the project is, the stakeholders have different status requirements. In an example below, i have listed a few questions that the different project stakeholders might have at Test execution phase.

Phase: Test Execution

Development Manager:
- What is the incoming bug rate ? Is the bug find rate declining or increasing?
- Which components have the most high severity bugs?
- Which components have the bugs that are blocking the testing?

Project Manager:
- Is the testing currently happening on track?
- Are there any major blockers to testing?
- Is there any risk to meeting test deliverables and milestones?
- If there are risks, what are the plans to mitigate the same?

Product Manager:
- Are there any defects that need Product Management attention?
- What is the list of deferred defects?

Test Manager:
- What is the percentage of test coverage completed?
- What is the percentage of defects found based on Test methodology- Scripted testing vs. Exploratory testing?
- What is the reason if the defects are trending up?
- What is the defect fix rate?

The above list is definitely not an exhaustive list but gives some pointers to varying status needs. There may also be overlap of questions that different stake holders needs answer for. These questions vary as the context and the project phase changes but it makes report more effective if these are anticipated in advance.
It saves a lot of time in the meetings when the status report answers the questions appropriately.

Answer each question in an objective manner:
Once the list of questions are thought of and finalized, the next step is to design the metrics. Pick up each question, assess each with the help of the data. It is important that answer to each question should be as fact based as possible. Refrain from adding emotional responses to the questions.

Position most important information at the beginning of the report:
The positioning of data in the Test status report is quite vital. Sometimes, i have observed the status report begin with all the positive information and has the risks listed somewhere at the bottom of the report. Such an organization of the report may cause oversight of important information. Effective status report should alert the recipients with any important information within one glance.
It’s always good to Summarize the status at the beginning of the report before getting into the details section.

Don’t get intimidated with mentioning the Risk items-
Test teams should not fear giving bad news. If the project is in the bad state, it should stated in a way that everyone understands the risks and the situation properly. Use forceful language if needed to pass the message properly but the language used should be professional.

Any more thoughts ?


Sharath Byregowda said...


I like the way you write your articles, it's very clear and easy to understand. I am still sharping my skills on writing better articles.

"Designing effective Test status reports" - does it really help?

Let's analyze the very first question you have posted "- What is the incoming bug rate ? Is the bug find rate declining or increasing?" - how does it help the developer? what could be the side effects of this on testing team and developement team? Can you think on those terms?

Anuj Magazine said...

Hi Sharath,
Thanks for your words.
It would have been great had you mentioned the context of your questions. Nevertheless, i will try and answer based on my thinking.

What is the incoming bug rate ? Is the bug find rate declining or increasing?" - how does it help the developer?

Incoming bug rate is a definite indicator of the stability of the module being tested. Having said this, just a mention of number of bugs would be an incomplete indicator of stability. Other factors that should invariably be considered along with number of bugs are Severity, Priority (if defined), Which component the bug was logged, whether the bug is valid or not, whether the bug is a regression issue or not etc. This information does help the developer to assess the stability of module and in some cases to forecast the work in coming time.

what could be the side effects of this on testing team and developement team?
The answer to this question depends upon the way the metrics are used in organization. If the metrics are blindly used to assess the performance of certain group, then the incoming bug rate may highlight the efforts of Test or development team in a positive way or negative way depending upon the numbers. This is obviously not the good practice.
In all cases, it depends upon the way the information is used. But if used in a right way, there are no side effects as such and the bug information can be a very useful indicator of healh of project (along with other relevant information such as Test coverage, resource utilization metrics etc.) and can provide useful insights in to risk mitigation.

I am not sure if i asnwered you fully but would like this discussion to continue if there are more questions.

Sharath Byregowda said...

@ Anuj,

What is the incoming bug rate ? Is the bug find rate declining or increasing?" - how does it help the developer? what could be the side effects of this on testing team and developement team? Can you think on those terms?

I feel bug rate is defined as the number of bugs discovered per week.

The side effects caused by bug curves is disccused in detail in BBST foundation class by Cem Kaner. To list a few
- Testers are made to find higher number of buugs early in the project, this gives them very less time to plan more complex, or even prepare and plan tests, which they might require during the later stages.
- Since more number of bugs are expected in the early stage of the project testers might find trivial bugs which might not have any value to the stake holder.

Couple of side effects when bug curve is expected to decrease

- testers are pressurised to run the same tests again and again.

- more bugs are categorised as not reproducible, invalid etc

Do you agree?

Anuj Magazine said...

I have used bug rates as bugs discovered per Test iterations, per builds as well in addition to weekly bug trends. If used with caution and with right kind of facts in place, bug trends can usually be quite a useful indicator of health of the project. I have seen the bug trends being used as Entry/Exit criteria for different milestones in the project.

The side effects of bug rates ends usually arise from the way the use of bug rates are communicated to the team. I was involved in one of the projects earlier on in my career where bug counts were given undue importance. One of the testers got an impression that logging huge number of defects means that one is a better tester. Going by this thought he logged thousands of defects in the product, most of which the developers didn’t even care to look at. All this definitely lead to lot of time being wasted in scrubbing the defects. Apart from that the loss that’s not measurable is the loss of credibility of not only the person logging defect mindlessly but also for the entire testing group for its immature processes. If i look back now, all this was caused by miscommunication from the superiors and not having enough checks in the process to avoid such instances when the mistake has been realized.
On the other hand, i have been a part of projects where the issue of Bug rate is handled with more sensitivity. Eventually the Bugs are one of the most important deliverables from the testing team. And logging the right kind of bugs at the right also forms one of the important goals and all this eventually takes the shape of bug rates. Yes, mindless slogging to increase the bug counts is of no use and proper checks must be in place in the process to prevent this from happening. And again caution must be taken in the way the use of bug rates are communicated to the teams that will be using these.
In my opinion, Defect migration statistics are more important to measure the effectiveness of a test group that the bug trends. This basically measures the defects were caught in the same phase (or build) in which they were introduced. To track the origin of defect is not a easy task and may require some effort in the beginning but if done in a right way.