A smoke test (in software) is a quick test done after a build has completed successfully, but before QA fully tests it.

Microsoft claims that after code reviews, smoke testing is the most cost effective method for identifying and fixing defects in software.

Because the sooner we find a defect, the cheaper it is to fix it, we use a continuous build process utilizing QA Wizard Pro to automatically perform a smoke test each time a build finishes. This lets developers know right away if something that they checked in recently caused a problem.

The process to implement an automated smoke test will vary depending on your application and the configuration of your build tool, but the basic steps should remain the same.

1. After the build has completed successfully, you may need to perform setup steps before testing your application. This may include copying files to the appropriate places, setting up database tables, installing licenses, or starting a server.
2. A second setup step is to get all of the QA Wizard Pro files required for your smoke test. We do this using Surround SCM’s command line tool to fetch several smoke test files to the local drive.
3. To maintain the most flexibility in the smoke test while keeping the build script static, we have a single script that is responsible for performing each test. We can then start the smoke test from our build tool with a line similar to:
   "C:\Program Files\Seapine\QA Wizard Pro\QAWRunScript.exe" "\SmokeTest\SmokeTest.qawwspace" "\SmokeTest\SmokeTest.qawscript" /Reportfile "\Build\CurrentReport.qawreport"
   The smoke test is then run and its report is saved with the rest of the build files. Should something fail in the smoke test, QAWRunScript.exe returns an error code of 1 which is picked up by the build tool and then reported to the developers along with the output of the script.
4. A final step is to clean up after the smoke test. This might include stopping a server, deleting files, or emptying database tables. This could also be done before the initial setup step to ensure that the environment is clean before any tests are started.

The smoke test itself contains a series of Script.CallScript() statements. Each script that gets called performs a different test. This is flexible in a couple of ways. It allows changes to the smoke test without the test developer needing to modify the build script. It also allows each individual test its own file, and possibly have its own creator so that work on creating these scripts can be shared.

When a task fails, most build tools will send the output of a task with the error report for that task. To take advantage of this, we use many PrintLn() statements in the smoke test scripts to detail which test is being run and what is happening at that stage in the test. This gives developers a better sequence of steps to be able to reproduce the defect that the smoke test found.

Related posts:

1. Data Driven Testing
2. What do I do?
3. Preventing Batch Files from Stopping Due to Individual Script Failure
4. Usability Test 1, Part 1
5. Usability Test, Part 2