One page test plan – contents
In yesterday's post I outlined when I use a one page test plan. In this post, we look at the contents of a simple test plan.
For most small releases, I have three big concerns:
- new functionality (what did we build for this release)
- old functionality (did we break what use to work by introducing new functionality)
- other quality criteria (as appropriate for the release - performance, security, usability, etc...)
For each of those areas, I'm interesting in the following areas of coverage:
- specific tickets (stories, feature requests, bugs, etc... in the release)
- general areas of functionality (either business facing or technology facing)
Ideally, for each area or ticket listed for each type of testing, there will be a brief description of the testing testing taking place along with a link to where I can find the details about the actual tests (a link into a test case management tool, a Google doc, etc...). Something that tells me at a high level what's going on and where I can find the details if I want them.
One page test plan
For small releases, I sometimes create what I call a One Page Test Plan. Using a One Page Test Plan assumes that the people involved are familiar with the technology, process, etc.... It's for small "boiler plate" releases (like a release to production after a two week sprint). That doesn't mean there's not complexity and risk, it just acknowledges that there are less documentation requirements since the team has a rhythm related to releases.
The contents of a One Page Test Plan are direct and straightforward. They assume the team is small and is in constant communication. Tomorrow's post will provide an outline of what's in a One Page Test Plan.
Let your bugs have social networks
One of the things I really like about JIRA is how much linking it allows. (Other tools do this too, but I wanted to namedrop the tool because they do it particularly well.) From a story, I can link related stories, defects, CM tickets, deployment tickets, etc.... Basically whatever ticket type I want. This is great, because over time I've developed some risk heuristics based on the number of links a ticket has:
- If it has a lot of links to other stories, I likely need to test more around business functionality concerns.
- If it has a lot of links to other bugs, I likely need to test more around technical functionality concerns.
- If it has a lot of links to CM tickets, I likely need to test more around deployment and configuration.
I've also developed some similar heuristics around estimating how long work will take based on links, how much documentation there will be to review, etc...
JIRA also shows you how many people have participated in a ticket. That is, it tracks who's "touched" it. I have similar heuristics around that. The more people involved, the longer stuff will take, the more likely there was code integration, etc...
What does the social network of your tickets tell you about your testing?
What’s in a smoke test?
When figuring out what to smoke test for a release/build/environment, I run down the following list in my head:
- What's changed in this build/release?
- What features/functions are most frequently used?
- What features/functions are most important or business critical to the client?
- Is there technical risk related to the deploy, environment, or a third-party service?
- Are there any automated tests I can leverage for any test ideas I came up with in the previous questions?
Based on my answers to those questions, I come up with my set of tests for the release. If it's a big release (aka more features or more visible), I'll do more smoke tests. It's been my experience that different releases often require different levels of smoke testing.
Calculating velocity
Catherine Powell had a great post yesterday on calculating velocity. From that post:
"So now we know that each QA engineer can do about 2.5 units of estimated work each week. When we go into the next estimation session, that's where we'll draw the line for test work. We estimate just like we always do, and we then will walk down the list committing to 2.5 units of work per week. When we run out of allotted time, we'll stop."
There are a couple of great tips in that post, and the overall focus on developing a method to calculate velocity if well done.
Finding the magic in your magic numbers
Today's tip comes from a post I read by Scott Berkun:
The simplest, sanest step in the world, a step few people do, is when a project ends go back and review your estimates and compare against the reality. Put it up on a big whiteboard, sit down with a handful of your team leaders, and discuss two things: what factors explain the difference, and what smarter things you might have done to have a better schedule.
That's an excerpt of his recent post on Magic Numbers of Project Management. Read the post for full details on the tip, but follow the advice. I suspect you're asked to estimate your work on a regular basis, and understanding where the magic comes from is an important part of that task.
Understand how you’re going to approach your testing
I do a lot of exploratory testing. So when I'm doing test planning, you'd think the "Approach" section of my test plan would be the shortest, right? Wrong...
I see a lot of value in thinking about, describing, and writing out how I'll approach my testing. So much so, that when I'm getting ready to execute what I think will be a particularly challenging charter, I'll take a few minutes to outline how I'm going to approach my testing. For some tasks, I might even write a short procedure so I don't mess something up if there's a factor or variable I want to control for while I'm testing.
My rule is that I should always be able to articulate how I'm approaching the problem. If I can't do that, I've got no business getting started with my testing. It means I've got some additional research to do before I'm ready. If I can outline what I'm going to do, than I'm ready.
Digging into a project
I recently started (another) new project. I find that whenever I start a new project, there are a few places I begin digging in:
- finding whatever documentation I can (requirements, diagrams, contracts, project plans, emails, meeting notes, etc...) - it's a mad dash to get anything that's been written down about the project
- finding (or making) a list of who's doing what - many times new projects come with new faces and names, keeping everyone straight can take some effort early on
- starting a new notebook or electronic notes file - I write down my open questions, my assumptions, and my ideas around testing/design/time-lines
When a new project gets dumped on your lap, where do you start first?
Is your testing process this clear?
In my opinion, one of the biggest success factors for a centralized testing organziation is a clear engagement model. One that starts with new project intake and ends with project closeout, follow-up, and portfolio management. I've tried to tackle this problem at two different organizations, and at each one I struggled to get clear and easy to follow definition around what we did.
Take a look at how HUGE laid out their process. I understand that it's marketing material, so it's obviously going to be pretty, but look past that. They lay out seven clear phases for engagement. Each phase has a summary of the steps that will be undertaken in that phase.
If you're centralized group offers products and services, imagine that your products and services are listed next to those phases. Those are the various products and services teams could expect during each phase. I find it to be a very simple visualization for what your organization might offer to project teams.
Don’t forget that some tests are open ended
A lot of testing literature talks about inputs and expected results. And that's all well and good, but you can't forget that there are multiple reasons to test and multiple ways to test, and that for some of them you can't accurately predict expected results. I find that a lot of early tests that happen in a project are focused on expected results. "We built the system to do X, does it do it?" However, over time the nature of the testing changes. It becomes more "what if" focused. Examples include, "How many users can we support on our current production configuration?" or "What happens to our users when that batch process runs?" or "What kind of errors might we see in the logs on a typical day in production?"
