Shall we test it?

by Giuseppe Lanzi on 02/24/2015

One of the most common questions people ask me during consulting sessions is “what do you recommend for testing?”. It’s a very interesting question that I find important, and it should be a primary concern for people beginning a new project. In fact, application tests are the only thing that can guarantee the end user a pleasant user experience.

To answer this question, I refer to cases analyzed in our technical support work, because the quantity and diversity of the applications we’ve studied is a good sample for understanding the nature of the most frequent errors.

I suggest these three rules:

The developer can’t run the test
Obviously I don’t mean that developers should do their jobs without testing what they’ve made, but rather that the developer’s test should only be preliminary: a test that must be passed before handing over the functionality to someone else who has not worked on developing that part of the application. This is a very significant consideration, because the person who implemented a form will use it according to their own way of thinking, which is the same way of thinking that led to the current implementation. It’s truly difficult to step off your own rails. It’s no accident that application testing is a career in itself.

Whoever does the internal test mustn’t receive instructions
This means that the colleague who tests the functionality should not get an explanation of how the app works, because they need to test it following their own intuition. In fact, often a problem that seems impossible to replicate locally can be replicated with help from a support tech. This is because when you already know how the application should be used, you don’t veer outside your pre-established patterns, and in practice you always click in the right place at the right time. But as soon as the application is used by end users, who have no pre-set mental pathways, the problem crops up.

Implement the beta test
When you put the application into production, even if you’re careful to follow the previous steps, problems that are less evident and not as easy to replicate always occur. In most of the cases we’ve seen in support, it took at least 2-3 causes at the same time. One possible solution is to install two versions of the application: the latest stable version without debug, and the version with the latest changes with debug activated. Afterward, by identifying a group of end users you can ask to use the version in debug, most users will use the stable version, while the beta tester group is already aware that they may run into small problems.

What do you think? What's the best policy to use for testing?

  •    You can't test thoroughly without relying on an outside service.
  •    You need an internal team dedicated strictly to product testing.
  •    Both of the above.
  •    None of the above (enter your comment).
Loading ... Loading ...

{ 5 comments… read them below or add one }

1 Ted Giles 02/24/2015 at 6:10 PM

Oh boy, is testing a can of worms? You bet your boots it is!

The testing situations I often find myself in are these;
1) The application is a shrink wrapped item which cannot easily be tested for bugs – like Excel as an example. These are subject to functional testing to see if the application does what the user expects/needs/wants. In the end, you get what you have bought so you probably need to adapt. Testing is almost always carried out by developers and professional testers.
2) The application is proprietary in as much as there are many users, each with a tweak here and there. There is often a road-map showing planned developments over a number of months/years etc and each of these will need to be tested at developer level, where regression testing will ensure nothing is broken. At the user level, new functionality must be tested as well as regression testing in case your particular version is broken.
3) Bespoke applications developed in house are often the most problematic and the approach I have taken with these is to have well documented test scripts, culminating in a GOLDEN SCRIPT which tests end to end functionality and integrity. Different people have different levels of testing interest. In my current role, one side wish to test data uploads and these lock out a different set of users. In isolation both parts work well, but as a whole it’s a disaster, (we are working on solutions with the software supplier). That’s why I adopt the golden script approach.

Getting over to business users that it is their responsibility to thoroughly test and re-test is difficult, and the only response I have come up with is “it’s your data, your business, and your money at risk if you don’t test properly”. Sometimes I win, sometimes I loose, but I have done my job.

2 Ted Giles 02/24/2015 at 6:14 PM

I wish I could spell lose as opposed to loose.
The first is something mislaid, the second is something flapping about.

3 John Talbott 02/24/2015 at 8:07 PM

In my experience .. there are multiple levels of testing but doesn’t require an outside service. The are of most importance is having a testing process that is integrated with the development process. Things tends to break down when the client, business analysts teams, testing teams and development teams are isolated from one another. This can be accomplished with cross-functional teams.

In an agile or any “short-cycle” iterative development process, a business analyst / tester can create functional test cases for a particular requirement/user story prior that can be review with the developer prior to starting development.

It aids in making sure all parties are on the same page prior to the development starting. Frequent check in sessions along the way until the requirement is deemed “ready for testing” results in fewer bugs when the “official” testing is to start. A seasoned carpenter might phrase it as “measure twice .. cut once.”

If the previously written and exploratory (not scripted) TCs do not fail, the requirement can be called “done” and be handed over to the client for User Acceptance Testing (UAT). UAT is a for the most part a non-scripted exercise.

This can be expanded to incorporate alpha and/or beta testing as it makes sense.

4 John Willems 02/25/2015 at 3:28 PM

Your section about the beta tester not receiving instruction really struck home with me. Great concept and excellent point. I am keeping that concept in the forefront as I work on my current app. I am at the stage where I am fitting the pieces together and starting to create the workflow. The timing of this post could not have better.


5 Giuseppe Lanzi 02/26/2015 at 5:24 PM

Seems like the tests are really interesting stuff for you guys, really a nice thing to see 🙂

I’m glad this post was good timed.

Leave a Comment

Previous post:

Next post: