Template for session notes
When I teach people how to do exploratory testing, a common point of confusion is around what to put in your notes. While I often tell people it depends on you, what you're testing, and the company you're working for - they still want some concrete advice. So I often show some examples from past projects and provide the following template:
- Mission: list out what you're testing with this charter
- Environment: list out meta information related to your test environment (versions, configuration, location, etc...)
- Risk: as you test, list out what risks you're looking for while testing
- Coverage: as you test, list out what areas, features, users, data, or other meaningful dimension you're covering while testing -- it's worth noting, I also instruct them to list out what they didn't have time to cover...
- Techniques: as you test, list out what you're doing... what techniques you're using, how you develop tests, etc... (in math class, this would be the "show your work" section of the document)
- Status: as you test, list out questions that occur to you that you need to get answered later, possible issues/problems/bugs you find while testing, notes about automation or future test sessions, etc....
- Obstacles: as you test, list out things that get it your way or ideas you have for things that would make your testing more effective -- this can be tools, hardware, information, training, etc...
For those readers who do session based testing on a regular basis, you'll notice I don't capture some of the classic items like setup time and time spent investigating issues. If you need to capture those metrics (or other metrics your team uses), simply add them in. Over time your session notes morph to become your own and you'll develop a format that works for you.
I'd be interested to see what other people capture.
XNote Stopwatch
When doing exploratory testing, I like to have a stopwatch running so I can keep track of where I'm at in my session. One of the most common tools I use for this is XNote Stopwatch. There are some cool features like "Always On Top" and "Transparency" that make is especially useful when testing.
Trick for clarifying a test charter
Sometimes when I ask someone what their test charter is, I get a paragraph in response. That's not bad, but I find that it often leads to a poorly understood scope definition, which leads to lack of focus while testing, which leads to a charter that runs way too long or feels unproductive. I have a trick I use to help simplify the mission of the charter when this happens.
Try using the following template:
"My mission is to test <insert risk here> for <insert coverage here>."
Some examples:
- My mission is to test for various boundary errors for Microsoft Word's bullets and numbering feature.
- My mission is to test for accurate error messaging pop-ups for Ford Motor Vehicle's Build and Price website.
- My mission is to test for SQL injection vulnerabilities for application login and administration screens.
- etc...
You might then still use the original paragraph to help detail out the charter, but getting a clear and concise mission helps me better estimate how much time I'll need to test, and maintain better focus while testing.
Thumb vote for priority
Sometimes, when reviewing charters with the team, I look to them to help provide some insight into what we should be focused on with our testing. One technique is to walk your list of charters (all twenty or them, or all 200) and have each person provide some insight into where they think it falls in priority. To facilitate this, I sometimes use a thumb vote.
When thumb voting, everyone must vote. There is no sideline. Since I usually use three levels of charter priority (A, B, and C), those map as follows:
- thumb up = A
- thumb sideways = B
- thumb down = C
What I find is that for most charters, most people on the team are on the same page (or really close). Every now and then, you have to make a decision on a close tie (but since you're the test lead, you're use to doing that anyway). Sometimes however, you see some odd votes. Like four up and four down. Those votes normally lead to some really great conversations. Often, people misunderstand the charter or have a different understanding of the risk involved. This surfaces those differences.
It's also fun. (In an I'm-a-tester-so-voting-on-charters-is-fun sort of way.)
Wrapping up a session
When I'm doing a 45 minute test session, I typically find that the last five minutes are reserved for wrap-up. Since I'm normally working with a timer of some sort, I'm normally aware of when I have five minutes to go. Typical activities for wrap-up include:
- finish what test or activity I'm in the middle of
- write down what I didn't get to that I think I want to come back to
- double check to make sure I have enough information for the bugs I need to log
- double check to make sure I captured all the relevant notes on test data for my session notes (which might mean saving a spreadsheet or something)
- clean up my test environment (if needed)
- stop my screen recording and save it off (if running)
- write down a couple notes about how I feel (did I need something I didn't have, energy level, frustrations, etc...)
It looks like a lot, but most of the time it's about five minutes. If it runs over, that's fine of course. Technically my hands-on testing is done when I'm finished with the first bullet point. But I like to include all the other stuff in my time estimates (aka my session time) because I feel it's all important for the work I'm doing.
Discipline in Exploratory Testing
Some people when you say you do exploratory testing immediately think ad-hoc testing. I suppose because there is less emphasis on obvious structure and at the end there is little tangible evidence of testing performed.
But in my view, there's a lot more to exploratory testing than wandering aimlessly through an application looking for bugs. As well as mentally challenging, it requires a lot of self-discipline.
Here's why you need self discipline:
1) You need self-discipline to test the parts that are not as interesting to you, or not as fun. It's easy to overlook and 'forget' them when other parts are more appealing.
2) You need self discipline to give each bug the time it deserves before racing off to find new ones. Time to analyze, examine and understand. Only then, can you go and look for new bugs.
3) You need self-discipline to write up bugs when they are found, instead of leaving them until later or when you feel like it.
In my view, in exploratory testing, as in many other ways of testing, its the mission and the stakeholder that count and their needs must come first.
What's different is that instead of relying on documents and reports, you need discipline to make sure you meet those goals.
Questions to help clarify test status
Stealing from a post I did on SearchSoftwareQuality.com, here are some questions I use to clarify testing status when doing debriefs:
- What was your mission for this session?
- What did you test and what did you find?
- What did you not test (and why)?
- How does your testing affect the remaining testing for the project? Do we need to add new charters or re-prioritize the remaining work?
- Is there anything you could have had that would have made your testing go faster or might have made your job easier?
- How do you feel about your testing?
Understand how you’re going to approach your testing
I do a lot of exploratory testing. So when I'm doing test planning, you'd think the "Approach" section of my test plan would be the shortest, right? Wrong...
I see a lot of value in thinking about, describing, and writing out how I'll approach my testing. So much so, that when I'm getting ready to execute what I think will be a particularly challenging charter, I'll take a few minutes to outline how I'm going to approach my testing. For some tasks, I might even write a short procedure so I don't mess something up if there's a factor or variable I want to control for while I'm testing.
My rule is that I should always be able to articulate how I'm approaching the problem. If I can't do that, I've got no business getting started with my testing. It means I've got some additional research to do before I'm ready. If I can outline what I'm going to do, than I'm ready.
Clarifying your charter
When I'm teaching exploratory testing, I find that one of the most difficult things to learn can be chartering. If you're not practiced at moving from the abstract to the specific it can be very difficult. It becomes even more difficult to figure out what will actually fit in your session time box.
Here are some tips for clarifying the purpose of your testing:
- Don't feel like you need to develop all your charters in one sitting or all of them upfront. Be comfortable with charters emerging slowly over time. While you'll need some charters defined upfront so you can get started, often you'll find that you charter-base will fill in as you go.
- Take the time to state the mission (or purpose) of the charter as clearly as possible. Don't say "Test the portal for reporting accuracy" when you can instead say "Test reports X, Y, and Z for errors related to start and end time selection criteria, summing/totally, and rounding." In my experience, the more specific you are, the better your testing will be. If you need to take an extra 30 to 120 seconds to get more specific, take them.
- Similar to the last tip, if you can't tell one mission from another, you've not done a good enough job defining it. If you have charters to "Test feature X," "Stress test feature X," or "Performance test feature X" can you tell me what the differences are between tests? Couldn't some stress tests be a subset of simple feature testing? Couldn't some performance tests be a subset of stress testing? If you can't compare two missions side-by-side and have clear and distinct test cases come to mind, than you might benefit from spending a little bit more time refining your missions.
- Finally, while you're testing go back and make sure you're mission is still correct. There's two goals here. First, you want to make sure you're on mission. If you need to test for rounding errors in reporting, but you find you just can't stop testing filters and sorting, than create the charter for testing filters and sorting and execute that charter instead. You can always go back to the charter for testing for rounding errors. Second, if you find as you test that you can better clarify your original mission, add that clarity as you go. It will help you when you go back to write new charters. The more clear you can make it, the easier it will be to recall what you actually tested three days later when you're looking back and trying to remember what work you still have in front of you.
Offsite Exploratory Testing Guidelines to Bug Reporting
Because it's such a good post, I'm stealing Anne-Marie Charrett's post from yesterday on "Do your bugs only glow when its dark?" In the post she outlines some guidelines she uses to report bugs when doing offsite exploratory testing:
- Provide the time it takes you to write up the defects in your estimate
- Get agreement on the level of detail for defect reports
- Write up the bugs as you find them, don't let them queue up
- Encourage customers to use a defect tracking tool
Read her full post for the details!
