Reference Page: Mnemonics in Testing
Today's tip is a content pointer.
Testing heuristics, tricks, and oracles, packed into mnemonics, on the reference page created and maintained by Lynn McKee.
Should I run that test?
Rick Grey brought the following to my attention as a Quick Testing Tip re-post candidate. It comes from Rikard Edgren over on thoughts from the test eye. Rikard takes something intuitive and builds a very direct heuristic around it in the form of a decision tree.
Attacking Passwords
Today's tip comes from Santhosh Tuppad. In his blog post "Password – Flaws / Risks / Design suggestions and more" Santhosh demonstrates the power of heuristic-based testing approach combined with tool-assisted tricks.
What you'll find is a friendly walk-through, comprehensive test and UX report, references for further learning, and "is there a problem" sample questions, - all in one.
A must read.
Keyboard shortcuts
With an N-tier applications testers may have to go under the Web presentation layer. For example, that could be a configuration program running on Windows Desktop. Or you may have to work with UNIX programs, directly or through a terminal emulator program. Some UNIX programs are simply called from the command line. Some other may have a sophisticated user interface, with "windows", "menus", "pop-up dialogs", and keyboard shortcuts. Today's tip is about the latter.
Track the "life span" of keyboard shortcuts to see if they are enabled/disabled on timely manner. Because, if they're not, you'll be able to bring a program into a really weird state.
As for Windows Desktop applications, track if keyboard shortcuts are defined consistently across different modules, and do not override standard meanings. That is, if "Quick Save" command is "Ctrl-S" in one module, it should not be "Alt-S" in another.
A Fly in the Ointment
Frankly, I picked up the idea of this technique after reviewing source code validating some input. I remember, programmers even called it "cheating". But since then, I successfully tried it on different applications without having an access to the source code - so, obviously, there are some patterns.
The idea
User data traverse through a few validation layers. What is valid at one layer, might be invalid for another. Data also get transformed and fed into other modules which may or may not have comprehensive validation of internal input.
The idiom
"A fly in the ointment" means a small defect that spoils something valuable or is a source of annoyance while being even in a tiny proportion.
The heuristic
Creating combinations of valid and invalid data sometimes allows passing through, or triggers a program to transform data into something causing problems internally.
A few examples in today's tip.
Some "restricted" characters
- Backslash (\). This character is used to "escape" other system characters, and to create system commands as well.
- Less than (<), Greater than (>), Ampersand (&). These characters have a primary meaning as tags in mark-up languages.
- Space character.
- Asterisk (*) is used as a wildcard in queries and regular expressions.
Some combinations to try
- Valid inputs wrapped up by tag characters. Examples: "<123>", "</123>"
- Restricted characters "escaped" with a backslash. Examples: "\&", "\\"
- System commands created with a backslash. Examples: "\d", "\t"
- Asterisk alone or in combination with a valid input. Examples: "*", "Toronto*"
- Space characters before, after, or around delimiters. Examples: " 123", "1. 23"
Armed with the examples provided above I went on the hunt and picked a couple of publicly open web-sites belonging to large organizations...
You can see results below.
Usability study: beliefs are more important than requirements
Have you ever heard "no one would do that"..."every user knows that" types of answers? You knew it's wrong. You've dealt with it.
And now here's a great article by Jakob Nielsen in today's tip: Mental Models.
Please read the entire article via the link provided, I put here just a few lines (per author's copyright requirements).
- What users believe they know about a UI strongly impacts how they use it.
- Individual users each have their own mental model.
- Many of the usability problems we observe in studies stem from users having mixed-up mental models that confuse different parts of the system.
Before I thought of "role-based testing" in terms of user access type roles and security settings. I see now how to enrich my exploratory testing scenarios based on mental model simulation: a newbie user, a "quick-clicker", a multitasker, came-from-competitor user,... list goes on.
Share your experience with us.
Go where the website suggests
In the good old days of the Web 1.0 world, when something didn't go well you'd get the basic 404. On a fancy site, that 404 would perhaps be branded. In today's Web 2.0 world, 404 pages are often smart. They try to figure out what you were doing and suggest places where you might go next. It's a cool feature, and a great way to take what might have been a poor user experience and turn it into something positive.
However, it doesn't always work the way we intended. When sites suggest things to you, they can make mistakes. Therefore, I have a heuristic that I follow which says always go where the software suggests I go. Here's an example...
Last week while playing around with the new mint data website, I was greeted with the "City not found" page several times.
A feature of this page is that it suggests a page for your to visit next. Often, it's a state. However at point point I was able to get the site to suggest I view spending for the entire USA. Using my heuristic, I followed the link and was greeted with an excellent java exception.
Rewarded with this exception, I was able to determine that the site is written using Java (sprint and hibernate), runs on Apache (and the version of Apache). I also have suspicion of what to do next. After seeing this issue I started trying custom URLs and was able to get several different exceptions. Some of which gave me additional insight into the site structure and possible test ideas.
Test for “ubiquitous” features
This morning I saw mint data for the first time. I love playing with software like this - this is cool stuff. Within a couple of searches (less then five) I found a couple of different issues I'd call bugs. All of them were found using a technique I call testing for "ubiquitous" features.
A ubiquitous feature is one that exists "everywhere." You don't question if it's actually a feature, you just assume it is. An example is using quotes in a search field. After using search engines for the last ten years, you just have some basic assumptions around how search works.
So this morning on mint data, I tested with quotes in my search. Once I performed that search, the interface completely froze up. I had to reload the site to get and new search to work. It was only later that I noticed that this feature is even illustrated in the mint data example search text.
Other search features caused similar issues. When you think about the application you're testing, it's sometimes useful to understand how users will map their existing expectations onto your application (which they likely don't understand yet) and how that will drive what they will assume it will do.
18 automation refactoring heuristics with code examples
Creating automation scripts from small steps and simple code is a common recommendation. In the context of learning programming, automation tools and object models it might even be considered a best practice. Yet practically using such scripts in testing brings minimal value due to variety of issues. Some of them related to programming.
Whenever encountering an external call that might break execution or raise exception see if you can improve it with error-handling.
On Error Resume Next boolRC = objRegExp.Test(sSource) intRC = Err.Number On Error GoTo 0 If intRC <> 0 Then boolRC = FALSE If Not boolRC Then Set objRegExp = Nothing Regex_Match = "" Exit Function End If
Suggested collection of refactoring heuristics covers levels of code block, subroutine, simple class, function library, and intended to help improving code, created with such tools as Quick Test Professional and Test Complete.
