Granularity of testing estimates
Yesterday I was discussing how I do testing estimates with a peer here in Indianapolis. He pointed out that testers in his organization estimate things in days and hours:
- "That's half a day."
- "That will take a day."
- "That's likely 200 hours."
His concern with this technique, was that people didn't really understand where their estimates were coming from. In some cases, when he would ask how they arrived at their number, some testers didn't know. They just guessed.
Listening to him, I noticed that I have a pattern when I manage teams. Early on, when I'm working with a new team or people I've not worked with before, I look for estimates in minutes. For anything other than sessions, I want to see the breakdown of where the time's going in minutes. It's only after I've seen you estimate a couple of times that I'll take hours and days. This is because once I trust that you have a model that you're using to estimate, you no longer need to "show your work." If you don't have a model, then I'll help you develop one.
Test ideas that come from test automation
I know it's not fashionable to like GUI-level test automation any longer. But whatever, I still like it. I'm unfashionable in more ways than one. I still like GUI-level automation for reproducing bugs, automated acceptance testing, and to support my performance and exploratory testing. I also like non-GUI tests, but I've never disliked GUI automation.
One reason I still write GUI-level test automation is because it helps me learn about the product. I'm still amazed at how many times I say "Wow, really?" when I'm writing my tests. Because I'm always picking at the GUI with tools like FireBug and Web Developer while I'm testing, I'm seeing things I don't normally see when I'm just clicking around.
For example, today I noticed:
- One of the applications I'm testing doesn't remove fields from the screen when they aren't active, it just hides them. I had never noticed until I counted on it not being there in my code.
- One of the applications I'm testing sometimes shows a parent child relationship using icons, and sometime doesn't. I had never noticed until I coded a rule expecting it to always be there.
- One of the applications I'm testing appears to have a relationship between fields that I wouldn't have expected. I discovered this based on field naming conventions.
All of these give me new test ideas completely unrelated to my automated tests. Anytime I'm surprised, I use that as an indicator that I have more tests to run. When I automate at the GUI-level, I often get surprised. I rarely get surprised when I'm automating against an API.
Watir
I've neglected mentioning Watir on this blog because it's already a well known tool. However, since I recently had to do an install of Watir yesterday on a new computer, I noticed a couple of new things the Watir community has been up to since last I checked in with them. For those who don't know, Watir stands for Web Application Testing in Ruby and it's a very simple tool (dare I say painless) for generating test automation for web applications.
So what's new? Well, the last time I installed Watir (it was a while ago), it only had (working) distributions for IE and FireFox. Now it appears to support almost everything I'd care to test: IE, FireFox, Safari, Chrome, Flash, and work is underway on other ports. I also stumbled across Celerity, a headless Java browser with JavaScript support. Mmmmm.... sounds fast.
I like Watir for a couple of reasons. First, I enjoy programming in Ruby - it's strangely relaxing to me. That Watir leverages my language of choice is a big win. Second, Watir is so easy to use. Between IRB and FireBug, there are few web applications I've encountered that I can't have working Watir scripts for in a few minutes. Watir commands are predictable and relatively few - making them easy to remember. Finally, it's very easy for me to extend other scripting tasks using Watir. Because it's Ruby, I don't just use Watir for test automation. I sometimes use it to drive test data entry, assist with exploratory testing, parse websites for data, and other odd tasks.
Another fun fact, Watir is one of the few tools with it's own podcast. If you do a lot of test automation, it's well worth taking the time to listen to the podcasts. Lots of gems in there, and not all of them related to Watir.
Added Bytes Cheat Sheets
If you're not familiar with them already, take some time to checkout the free cheat sheets on the Added Bytes website - they are a set of quick references for a variety of languages and technologies. I use the Regular Expression, Ruby, and SQL cheat sheets quite a bit. I'd like to use the Subversion and Rails sheets more, but just haven't had the opportunity.
soapUI
I'm surprised I haven't posted about soapUI yet given how often I use it. It's one of the first web service testing tools I ever used, and I haven't had much need to look elsewhere. soapUI is great for both functional testing and performance testing. They also have a great service mocking feature.
I've never used the pro version, so I can't speak to those features or the support that comes with it. But I will tell you this, the first (and only) time I had an issue with the tool I contacted the developers and they had a build for me the next day with a fix. I was amazed at the turnaround. I'm sure not all fixes can happen that fast, but it still says something about their passion for the tool and the people that use it.
You can checkout the full feature set here: http://www.soapui.org/features.html
The performance pitch
Stoyan Stefanov recently posted a fantastic set of slides on pitching better website performance. The slides look at different websites and how performance affected the core business. It's a cool idea. I think material like this is useful for a variety of reasons, but most of all because it provides insights into how I can better transform my technical work into something that talks about business value. Each slide is a mini case study on looking at a company and translating performance into a business driver important to them. As testers, we need to be good at that. The more exposure we get to materials like this, the easier it becomes to recognize similar opportunities in our own organizations.
Identifying boundaries
Boundary value analysis is one of the most popularized testing techniques out there, and rightly so. It yields results. Most people think of a simple example when they think of boundary value analysis; like a field with a maximum character limit of 20 characters and visible character limit of 10 characters. We know that the field has three explicit boundaries: 0, 10, and 20 characters.
What's often more interesting to me, are the other boundaries related to that field. For example, it's likely that whatever value I enter is going to get stored somewhere. The variable where it's stored has boundaries based on it's datatype. If it's stored to a database, the field has boundaries. If it's stored to a file, the input and output programs that read and write the value expect certain boundaries. If it's ever rendered on the screen, it's rendered based on certain assumptions about it's boundaries. And certainly any processing done using the value includes boundaries (whether the programmer was thinking about them or not).
All of this leads to some interesting analysis of what we really mean when we say we're doing boundary value analysis. We're really looking at all of those various places where a boundary might exist. We're also looking at the potential interations of those boundaries with other variables within the system, and often we won't know what those are right away. Recently, I tried to illustrate some of the complexities of this in an example I did using Microsoft Word. In that example, I tired to model the boundaries related to bullets and numbering.
Here are some of the high-level things I think about when doing this type of analysis:
- What boundaries can I see (explicit)?
- What boundaries can't I see (implicit)?
- What boundaries exist around persistence?
- What boundaries exist around business processes and interaction?
- What boundaries are shared with other variables?
- What boundaries can be modified or configured (meaning they're variable, not static)?
- What boundaries exist in relation to time (get reset, race conditions, etc...)?
- What external boundaries influence this boundary (like the size of a disk or size of a piece of printed paper)?
Ideas for stress testing
With most project teams I've worked with, I've found that when I say the words "stress testing" people really hear the words "load testing." To me this is rather odd, but when I step back and think about it, I guess it makes sense. Most applications I've worked on have been web applications, and in that world load is often one of the biggest factors related to stress.
However, to help remind myself that there are other forms of application stress, I use the following checklist when I evaluate what stress testing I want to do. For each of these, I look at each piece of the architecture (desktop/browser, database, service/server, etc...) and in turn ask myself, "What would happen if...?"
- heavy load:
- large number of users
- large volumes of data
- large number of processes
- large number of transactions
- constrained resources
- not enough memory, processing power, or disk space
- not enough bandwidth
- not enough queues (or even shallow queue depth)
- not enough licenses
- inability to access files, servers, or services
- unfriendly inputs
- invalid file types and/or data types
- fuzz testing
- very large (or very small) inputs
- shared resources
- crash or stress other applications in a shared server space or that use a shared device/service
- timing
- constant or steady load for long periods of time (endurance)
- intermittent loads with long pauses in between transactions (looking for timeouts or parts of the system that go to sleep or get deprioritized with inactivity)
- concurrency (trying to force potential race conditions, file/database locking, etc...)
Look for opportunities to shortcut a process
When I'm testing applications, I keep an eye out for opportunities to shortcut a process. For example, if you got to book a hotel room on Expedia, you very likely do the following:
- Enter an arrival date and end date
- Enter a city or address
- Change the default number of rooms and/or travelers
When you click search, you get a set of results back and all of them will have some indication or price and availability. I suspect that this is the "normal" process for booking on Expedia. However, you can also "repeat a trip" using Expedia. When you repeat a trip from your past itineraries, it defaults the hotel selection to the same one from the trip. You just select new dates.
Recently, I did a normal search for a hotel, and Expedia told me that the hotel was unavailable for my dates. When I repeated a previous trip for the same hotel, suddenly those dates were available and I could book. I found that interesting. Apparently, there are different rules that fire in each process flow.
When I'm evaluating what to test in an application, I look for those opportunities where I can shortcut one process with another. It's not uncommon to find differences in rules, available data, or messaging inconsistencies when doing so.
Do we really need a script or can we use a checklist?
On a current project, we have a lot of testing that requires deep domain knowledge. Given the level of domain knowledge, we're planning on getting domain experts to do a good portion of the testing. This has uncovered an interesting opportunity for us. To date, most testing done in this organization has been scripted (giant Word documents full of steps and screenshots). In the past I believe this was done because the testers didn't have the requisite domain knowledge to execute the tests without a lot of help. But now, by making this shift, we can reassess if we really need test scripts.
We've decided that for much of the testing, we'll switch to checklists instead of test scripts. While we might still have a couple of test procedure documents which provide high-level outlines for how to do something, the details of the testing will be in some (relatively) simple Excel checklists. This saves us a lot of work, since we don't need to update a bunch of artifacts. I suspect it will also increase our test execution velocity by reducing the overhead of running a series of tests.
Kaner has been talking a lot recently about the value of checklists. If you've not reviewed the work, check it out. Look for opportunities where you might leverage checklists instead of some of the more traditional heavyweight scripts associated with testing in IT corporate environments.
