Designing effective Test status reports

"You are the headlights of the project"- Sounds something familiar? Yes, it is the very first lesson of the landmark Software Testing book- "Lessons learnt in Software Testing". The intent of this lesson is simple- Testing is done to find information . This is infact the essence of testing group. The project stakeholders always rely on the results of the testing effort to get a feel of how the project is coming along. And it is this information that helps executives take important decisions about releasing and positioning the product under test in the market. That is probably one of the important reasons why organizations invest in an independent testing group.
There is no doubt that finding information is one of the premier role of the Testing group in an organization but what is equally important is how effectively the data is actually presented. I was reading an article on Metrics by Mr. L.R.V Ramanna and the below mentioned quote took my attention-

Test metrics are like corporate balance sheets. What they show is important but what they hide is vital.

So Test status, depending upon the way it’s presented- can help take Critical decisions with ease or rather than answering questions, raise more questions. Consider, a project is 2 day away from the release and the testing status report tells only about the Open defects and the defect trends and misses vital information such as Test coverage etc. may cause decisions to be made without enough considerations to the facts unless somebody raises the hidden questions

Here i am sharing some tips based on my experience on how the status reports can be made more effective.

Identify the audience of status report:
Knowing the audience who will read the report is an important exercise. More often, i have seen that the Test status reports are sent to distribution lists containing large number of people without a thought to who the recipients are. Having knowledge does not mean to know the people personally, which may not always be possible if you are working in an organization distributed world wide but it means to understand their role in the project. This always helps to anticipate the expectations people have from the status report.
It is always preferable to list the role of people receiving the status report which may be as diverse as the following-
- Product Manager
- Project Manager
- Development Manager
- Development Director
- Test Team
- Development team
- Vice President etc.

If the status reports are sent without knowing who is receiving it, then it often leads to majority of recipients ignoring the report upon receiving it and you take the blame of unnecessarily spamming their inboxes.

Anticipate what questions the recipients would have:
This step is the basis of Test status reports.
The key purpose of Test project status reports is that it answers the wide variety of status related questions that the project stakeholders might have. An effective status report would always answer the diverse questions clearly. Depending upon the phase the project is, the stakeholders have different status requirements. In an example below, i have listed a few questions that the different project stakeholders might have at Test execution phase.

Phase: Test Execution

Development Manager:
- What is the incoming bug rate ? Is the bug find rate declining or increasing?
- Which components have the most high severity bugs?
- Which components have the bugs that are blocking the testing?
etc.

Project Manager:
- Is the testing currently happening on track?
- Are there any major blockers to testing?
- Is there any risk to meeting test deliverables and milestones?
- If there are risks, what are the plans to mitigate the same?
etc.

Product Manager:
- Are there any defects that need Product Management attention?
- What is the list of deferred defects?
etc.

Test Manager:
- What is the percentage of test coverage completed?
- What is the percentage of defects found based on Test methodology- Scripted testing vs. Exploratory testing?
- What is the reason if the defects are trending up?
- What is the defect fix rate?
etc.

The above list is definitely not an exhaustive list but gives some pointers to varying status needs. There may also be overlap of questions that different stake holders needs answer for. These questions vary as the context and the project phase changes but it makes report more effective if these are anticipated in advance.
It saves a lot of time in the meetings when the status report answers the questions appropriately.

Answer each question in an objective manner:
Once the list of questions are thought of and finalized, the next step is to design the metrics. Pick up each question, assess each with the help of the data. It is important that answer to each question should be as fact based as possible. Refrain from adding emotional responses to the questions.

Position most important information at the beginning of the report:
The positioning of data in the Test status report is quite vital. Sometimes, i have observed the status report begin with all the positive information and has the risks listed somewhere at the bottom of the report. Such an organization of the report may cause oversight of important information. Effective status report should alert the recipients with any important information within one glance.
It’s always good to Summarize the status at the beginning of the report before getting into the details section.

Don’t get intimidated with mentioning the Risk items-
Test teams should not fear giving bad news. If the project is in the bad state, it should stated in a way that everyone understands the risks and the situation properly. Use forceful language if needed to pass the message properly but the language used should be professional.

Any more thoughts ?