Work backwards from a destination
Many of us are comfortable testing from a use case. It's familiar. Most of us are trained and experienced in thinking about a problem starting from an actor, who takes action, and ends up somewhere. It's the classic "Given... When... Then..." way of thinking about a problem. You start from point zero, then advance to the destination. Lately I've found it helpful to instead start from the destination, and then work backwards from there to think of all the different ways I could end up there.
For example, let's say I'm on a search results page. Here are all the ways I might have gotten there:
- I go to the search field on the homepage, enter search criteria, I land on the search results page
- I enter a URL someone sent me which takes me to the search results page
- I click on a link with a hard-coded search request that takes me to a search results page
- I click the back button on my browser which takes me to a search results page
- etc...
This technique is sometimes helpful when the piece of functionality I'm testing has preconceived notions about what defaults it can assume based on how the programmer thinks I got there. They were likely thinking of the one or two use cases they had available - not the 8 or 9 other ways you can get there.
For example, most features on search results page (like sorting, filtering, etc...) assume you've entered search criteria. In some cases, that's just not true. Sometimes you can get to a results page without entering anything. If your search criteria isn't pre-populated in the search field for you, in those cases you can get some really odd behaviors. This can become even more interesting when you're dealing with features that perform calculations, or where the issues resulting from the false assumptions aren't immediately visible.
The next time you run across a feature that looks like it might be making some assumptions based on state, work backwards from that point to think of all the other ways you might be able to get there. Do all of those paths lead to the necessary conditions for the defaults to be populated? If not, what possible problems might that cause?
A Fly in the Ointment
Frankly, I picked up the idea of this technique after reviewing source code validating some input. I remember, programmers even called it "cheating". But since then, I successfully tried it on different applications without having an access to the source code - so, obviously, there are some patterns.
The idea
User data traverse through a few validation layers. What is valid at one layer, might be invalid for another. Data also get transformed and fed into other modules which may or may not have comprehensive validation of internal input.
The idiom
"A fly in the ointment" means a small defect that spoils something valuable or is a source of annoyance while being even in a tiny proportion.
The heuristic
Creating combinations of valid and invalid data sometimes allows passing through, or triggers a program to transform data into something causing problems internally.
A few examples in today's tip.
Some "restricted" characters
- Backslash (\). This character is used to "escape" other system characters, and to create system commands as well.
- Less than (<), Greater than (>), Ampersand (&). These characters have a primary meaning as tags in mark-up languages.
- Space character.
- Asterisk (*) is used as a wildcard in queries and regular expressions.
Some combinations to try
- Valid inputs wrapped up by tag characters. Examples: "<123>", "</123>"
- Restricted characters "escaped" with a backslash. Examples: "\&", "\\"
- System commands created with a backslash. Examples: "\d", "\t"
- Asterisk alone or in combination with a valid input. Examples: "*", "Toronto*"
- Space characters before, after, or around delimiters. Examples: " 123", "1. 23"
Armed with the examples provided above I went on the hunt and picked a couple of publicly open web-sites belonging to large organizations...
You can see results below.
Test for “ubiquitous” features
This morning I saw mint data for the first time. I love playing with software like this - this is cool stuff. Within a couple of searches (less then five) I found a couple of different issues I'd call bugs. All of them were found using a technique I call testing for "ubiquitous" features.
A ubiquitous feature is one that exists "everywhere." You don't question if it's actually a feature, you just assume it is. An example is using quotes in a search field. After using search engines for the last ten years, you just have some basic assumptions around how search works.
So this morning on mint data, I tested with quotes in my search. Once I performed that search, the interface completely froze up. I had to reload the site to get and new search to work. It was only later that I noticed that this feature is even illustrated in the mint data example search text.
Other search features caused similar issues. When you think about the application you're testing, it's sometimes useful to understand how users will map their existing expectations onto your application (which they likely don't understand yet) and how that will drive what they will assume it will do.
Golden Scenarios
Today's tip comes from Brett Leonard.
In the Beginning... Mass Hysteria and Confusion...
When faced with a new testing challenge, finding a place to start designing your tests can be difficult. It is natural to suffer from stimulus overload and feel overwhelmed. Using a minimalist approach to data setup can accelerate the development and execution of complex scenarios and quickly provide a means to report on the status of the application. In this article I will describe a testing challenge I recently faced and how I used a simple end-to-end test scenario, or “Golden Scenario”, to rapidly test and track the development of the feature under test.
A Challenge with No Easy Answer
Recently, I needed to design tests for a complex feature that had been in development for a few months but had not been tested from an end users point of view. I needed to quickly understand the feature and design a method to test it. The first thing I did was schedule a meeting with our developers, architect, and product manager. Our meeting was productive but did not give me enough information to begin drafting my test scenarios - much of our conversation revolved around the data architecture challenges that were still ongoing.
I wanted to find a way to cut through the complexity and define initial scenarios. Once the initial scenarios were defined and communicated, I would have a mechanism to report on the application status by tracking the progress of our test execution.
Studying Economics Finally Pays Off
The next day, while contemplating screenshots of the feature, my schooling in economics came to mind. Economists use simple models to understand complex systems. For example, in my international economics class, we constructed a model using a country that produced one good - a ”one good country” and examined the supply and demand curve in this simple economy. Once we understood the “one good country” economy we added complexity by introducing another “one good country”. Later, we increased the number of goods and created much more complicated models. This led me to the idea of using a similar approach to develop my first end to end scenario that I deemed, the “Scenario of One”.
The Golden Scenario Technique is Born
The Golden Scenario Technique (ours was named “The Scenario of One”) uses a minimalist approach to setting up test data. To implement it you simply define a scenario that requires the absolute minimum amount of setup necessary to interact with your targeted feature. In my case, I’m testing a project management application, which starts with “projects” as a container. A “project” can have multiple “packages” and “packages” can have multiple “participants”.
Our First Scenario Defined and Communicated
My first scenario was to create a project with one package and one participant. The act of setting up the data for this scenario exercises a great deal of code that would provide us with valuable information about the state of the application. By defining and adding this scenario to our testing dashboard we finally had a clear direction and method to communicate test results.
Excited with our path forward, I quickly wrote up a mission in One Note© (my favorite test documentation tool), and presented it to my team. That morning in the daily development status meeting (or daily Scrum meeting), I described my idea and declared that our mission for the day would be to complete the setup of the “Scenario of One” or document the roadblocks that prevented it.
Mass Hysteria and Confusion Evaporates
With the mutual understanding of this limited scope I broke down the complexity to a manageable, understandable, and reportable form which I can now use to guide the development and execution of additional test scenarios. I had blazed a path forward and felt as though I had finally taken control of our testing effort.
To document our results in real-time, I drew a chart on our dashboard with the various steps of the process in a straight line. As we executed our tests, I would use color-coding to indicate how far our testing had progressed. At first, there was a lot of red but eventually (after two painful weeks) we were able to report that we made it through all variations of the scenario.
Jump Start Your Testing with Golden Scenarios
The next time you feel overwhelmed with a testing situation, try asking yourself “what is the minimum amount of set up that I can do to start testing this application?”, make it your first scenario, and begin testing. It will jump start your test execution and give you valuable information about the state of your application. You will be well on your way towards providing the valuable information testing is supposed to provide the stakeholders of your application.
Crash test your application
Image courtesy: theinspirationroom.com
Sometime ago I wrote about a quick way to shutdown a hanged application.
But, in fact, shutting down a working application could help you exposing certain types of resource management defects. This is how ends become means.
Sample tests, suggested below, were tried on different Windows Desktop applications, sometimes helping to reveal serious issues.
Local Tests
- Start an application. Begin working process then shutdown the application in the middle of it. Start the application. See if it reports than the previous session wasn't properly finished (and does it suggest something about it, like "start in safe mode").
- Start an application. Begin working process then shutdown the application in the middle of it. Start the application. See if it can access any previously opened files, and how is it handling and reporting the issues.
- Change settings in an application, apply changes. Shutdown the application. If the application saves settings only at exit point, all changes will be lost.
- Change settings in an application, apply changes and shutdown the application simultaneously. Start the application. See if it can access configuration files, does it report that they might be corrupted, does it suggest repairing them? (you can also try corrupting the files manually).
- If an application allows running 2 copies: open 2, then shutdown one of them. Observe behavior of the remaining one.
Server Tests
- Start a client. Shutdown a client. Try to login as the same user: a) immediately b) after a timeout given.
- Start a client. Shutdown a client. Repeat this process (you may want to automate that) while monitoring how Server side of an application allocates and releases (does it?) RAM and hard-drive resources.
- Start a client. Shutdown a client. Repeat this process (you may want to automate that) for a while - you may see errors like "max number of connections exceeded", etc.
Let us know results of your exploration.
Visualize XML then Blink Test
XML format is a very popular data container. Programs read, generate, convert, and exchange data in XML format.
XML data chunks can be uploaded from application's GUI, or can be generated by a client part and then transmitted to a server. XML data input can be effectively used when an application has unstable GUI or has no GUI at all.
<pre><?xml version="1.0" encoding="utf-8"?> <transactions> <debit> <amount>10.0</amount></debit> <credit> <amount>55.0</amount> </credit> <debit> <amount>120.0</amount> </debit> <debit> <amount>25.0</amount> </debit> <credit> <amount>5.0</amount> </credit> <credil> <amount>15.0</amount> </credil> <credit> <amount>15.0</amount> </credit> <debit> <amount>60.0</amount> </debit> <debit> <amount>50.05</amount> </debit> <debit> <amount>75.0</amount> </debit> </transactions></pre>
...But, in testing, XML data verification task means time-consuming process of going through countless, similar-looking text lines, with a high chance to get bored and miss an inconsistency.
Sample Task
Verify “transactions” xml file presented above.
Only “debit” or “credit” records are valid. Any other entries must be located and reported.
Additionally, identify and report all debit records with debit amount greater than 50.00.
As you can see, even going through just 10 records takes a significant time and might become really boring task.
With a 100 of records it's pretty much a brain-dead work to do.
How can we transform it to re-enable brain-powered testing?
Blink Testing technique, as described by James Bach, would be a perfect fit to engage rapid pattern recognition capabilities of a human brain. However, applying it directly to a textual source of an XML file may lack of efficiency due to complexity of a pattern.
Transforming XML text into a web-page looking table (HTML table) with inconsistencies color-coded, gives an instant productivity boost for blink testing.
XML visualization technique
XML transformation, presented above, is performed with help of XSL (Extensible Stylesheet Language) script.
Creation of XSL scripts does not require installing any IDE - you can use just a text editor, like Notepad. You would need minimal programming experience to start creating your own scripts, and you can easily modify existing templates.
XML source sample and its visualization were taken from my article "XML verification example".
You can also find there XSL source code template and links to other XML/XSL related articles.
Identifying boundaries
Boundary value analysis is one of the most popularized testing techniques out there, and rightly so. It yields results. Most people think of a simple example when they think of boundary value analysis; like a field with a maximum character limit of 20 characters and visible character limit of 10 characters. We know that the field has three explicit boundaries: 0, 10, and 20 characters.
What's often more interesting to me, are the other boundaries related to that field. For example, it's likely that whatever value I enter is going to get stored somewhere. The variable where it's stored has boundaries based on it's datatype. If it's stored to a database, the field has boundaries. If it's stored to a file, the input and output programs that read and write the value expect certain boundaries. If it's ever rendered on the screen, it's rendered based on certain assumptions about it's boundaries. And certainly any processing done using the value includes boundaries (whether the programmer was thinking about them or not).
All of this leads to some interesting analysis of what we really mean when we say we're doing boundary value analysis. We're really looking at all of those various places where a boundary might exist. We're also looking at the potential interations of those boundaries with other variables within the system, and often we won't know what those are right away. Recently, I tried to illustrate some of the complexities of this in an example I did using Microsoft Word. In that example, I tired to model the boundaries related to bullets and numbering.
Here are some of the high-level things I think about when doing this type of analysis:
- What boundaries can I see (explicit)?
- What boundaries can't I see (implicit)?
- What boundaries exist around persistence?
- What boundaries exist around business processes and interaction?
- What boundaries are shared with other variables?
- What boundaries can be modified or configured (meaning they're variable, not static)?
- What boundaries exist in relation to time (get reset, race conditions, etc...)?
- What external boundaries influence this boundary (like the size of a disk or size of a piece of printed paper)?
Look for opportunities to shortcut a process
When I'm testing applications, I keep an eye out for opportunities to shortcut a process. For example, if you got to book a hotel room on Expedia, you very likely do the following:
- Enter an arrival date and end date
- Enter a city or address
- Change the default number of rooms and/or travelers
When you click search, you get a set of results back and all of them will have some indication or price and availability. I suspect that this is the "normal" process for booking on Expedia. However, you can also "repeat a trip" using Expedia. When you repeat a trip from your past itineraries, it defaults the hotel selection to the same one from the trip. You just select new dates.
Recently, I did a normal search for a hotel, and Expedia told me that the hotel was unavailable for my dates. When I repeated a previous trip for the same hotel, suddenly those dates were available and I could book. I found that interesting. Apparently, there are different rules that fire in each process flow.
When I'm evaluating what to test in an application, I look for those opportunities where I can shortcut one process with another. It's not uncommon to find differences in rules, available data, or messaging inconsistencies when doing so.
Changing the testing culture in your organization
Many of you may have already watched the Google video about testing on the toilet, so forgive me if I bore you, but I truly found this video inspirational.
Its a story on changing the testing culture within an organization.
The key points that struck me where:
- The testers had a high profile stakeholder who championed and explored their cause.
- They didn't try to sell testing themselves, they looked for volunteers from all over organization to come up with ideas to change the culture
- They didn't try to apply a 'one approach fix all' to all groups. Instead new staff were trained up during orientation, and existing staff were trained through free talks
- When one approach didn't work, they changed it and tried something different. They never gave up.
There is much more in this entertaining video here's link to watch it yourself. Its about 3o mins in length.
Trick for clarifying a test charter
Sometimes when I ask someone what their test charter is, I get a paragraph in response. That's not bad, but I find that it often leads to a poorly understood scope definition, which leads to lack of focus while testing, which leads to a charter that runs way too long or feels unproductive. I have a trick I use to help simplify the mission of the charter when this happens.
Try using the following template:
"My mission is to test <insert risk here> for <insert coverage here>."
Some examples:
- My mission is to test for various boundary errors for Microsoft Word's bullets and numbering feature.
- My mission is to test for accurate error messaging pop-ups for Ford Motor Vehicle's Build and Price website.
- My mission is to test for SQL injection vulnerabilities for application login and administration screens.
- etc...
You might then still use the original paragraph to help detail out the charter, but getting a clear and concise mission helps me better estimate how much time I'll need to test, and maintain better focus while testing.
