QAGuild

Introduction

Software development is complicated. Organizations spend a large sum of money and man-hours into it. This, coupled with the fact that the shrinking of hardware cost, one would expect that the software produced would increase in quality every year. But as Chaos Report shows, our software industry is currently in crisis, with "84% of projects fail or are significantly challenged", and "45% of developed features are never used". Although Chaos Report does receive fair share of criticisms, but there is no denying the quality of software is purely perceived by public, to the extent that everyonewould expect software projects to delay and buggy.

With this, it's no wonder that a lot of companies actively invest a lot of time in software development practices, QAs, CMMIs and so on and so on. People are seeking for silver bullets, and one of such bullets proposed is automated functional testing. Software is complicated, the number of execution paths that the software can run through is close to infinity, thus It is hoped that by automating the manual test cases, the occurrence of regressions could be kept to minimum.

Unfortunately, a lot of past automation efforts ended in failure, partly due to the unrealistic expectations on what automated testing could do. In this article, I am going to explore the pitfalls of automated testings, and how to write effective test scripts. Hopefully it would be of some help to my readers.

Pitfalls

1.It's the Silver Bullet!

Sorry, but there is no silver bullet in software development. Automated testing included. Automated testing is just like any other parts of software development; where the success is dependent on the quality of the personnel involved and the management. Hoping that automated testing can work miracle is not a realistic, neither a helpful expectation.

2.Record and Playback solves all the scripting problem!

The tooling vendors love to emphasize how their tools can help you to write scripts by "record and playback". You, as a scripter, don't have to manually punch the keyboard to write scripts-- the tools can do it for you. All you have to do, is to use the automated tools to play around with the software, and viola, a test script is created! And you will just have to run this script every time your software is released!

There are several fronts why this is wrong. First, the objects that automating tools captured are always primitive, they can only capture the low level representation of the objects, sometimes even only coordinates ( In the recent years the tools have been improved to support higher level objects, such System.Windows.Forms object in .Net framework, but still). This kind of test scripts is brittle-- if your object is slightly off the coordinates, the test fails, even though it shouldn't. Second, even if the test tools can write a perfect script a human can write, we the human still have the specify what to expect.

3.Testers can be disposed!

Another myth concerning automated testing is that since testing tools can do the job, there is no need for manual testers, and hence they can be disposed. This is also wrong. Manual testers are still needed for the following task:

1.Someone needs to design and write test cases, and until we can build AI powerful enough to displace human, only manual testers can adequately do the task.

2.Human are more flexible than computers; while computers can only follow predefined steps, human can cover more test paths because we are more creative. We can do exploratory testing, and computers can't.

4.All tests can be automated, and must be automated.

100% test automation is good! So let's do it! Again, this is another blind optimism ( or dogmatism) that doomed many software projects to failure. There are tests that cannot be automated.

Take for example, how do you verify that the css rendering is OK ( whatever that may mean) across different browsers? Visual verification, as they say, is often beauty in the eyes of beholder.

And even if you can automate the tests, it's worthwhile to do a quick calculation to see whether those cases are worth automation or not. Instead of doing functional testing, what about isolating the point of fault and create an unit test instead? Running unit test is cheaper than running a full-blown regression test.

5.The ROI period is short.

Functional automated testing is a long term project. All the test scripts need time to write. So you have to make sure that the the time saved by outweighs the cost of writing

How to Benefit from Automated Testing

1. Unit tests for code coverage, automated tests for smoke tests.

Unit test is a powerful tool, if you want to do code coverage. It's quick, it's easy to write, and it can cover a lot of cases that usually can't be covered by automated testing ( think of Out Of Disk Exception). Not only that, the developers can use the favorite IDE to write the tests, and hence greater productivity. Automated testing, on the other hand, is for integration tests and verification tests.

About the author

Soon Hui is a .Net developer working in Esteem Innovation, a software company that creates desktop applications to help structural engineers in drafting, analysis, design and detailing