Software testing is a vast field with a variety of methods, tools, and terminology that can sometimes be confusing. For a team hoping to develop and invest in quality software, finding their way into this field and selecting the best practices could prove difficult. At Spindev, we can provide recommendations for how certain aspects of functional testing should be performed. After sharing some general information about testing, we will walk you through a concrete example using image analysis software.
For the purpose of this article, our Software Under Test (SUT) is an image analysis solution made of a Graphical User Interface (GUI) and an engine. The GUI is used to select input files, launch commands, and see results (e.g., data or ‘output’). The engine is the part of the software that is actually doing the computation. This is where the algorithms live and probably where most of the software engineering work is done.
What type of tests?
There are many verifiable properties in a product that can help assess the quality of the software. These properties include things such as features, performance, usability, security, and more. Depending on the product and its users, the level of expectations for each property differs… except for the feature coverage. A product should work according to its specifications or documentation. The functional tests are used to verify that this is the case. Meanwhile, the functional tests are grouped into different categories: unit tests, integration tests, component tests, and end-to-end tests to name a few.
When we order the different types of functional tests by the depth of the technical layer, then the two ends of that spectrum are well defined:
- Unit tests are done on a small unit of code, directly on the source code and check one low-level function of the program at a time.
- End-to-end tests verify that the business expectations of the product are met using the final/packaged version of the product.
In between, we find many other types of functional tests: integration tests, module tests, components tests, etc. However, the characteristics of those tests are unclear.
To make things clearer, we can look at the different types of functional tests another way. We can arrange them in two categories: black box and white box tests. White box tests are targeting the internal structure of an application and hence are performed on the source code (e.g. unit tests). Black box tests are checking the functionality and are performed on the final product or modules without knowledge or access to its internal structure.
Most often, white box tests are written and launched by developers. The tools and methods are well-known and do not depend much on the software under test. Black box tests can be written by developers, QA, or even product team members. These are the tests we will focus on. From here, we will follow our image analysis software example.
Let’s say our SUT can compute many characteristics of an image. For example, it can detect if an image is in color or black and white. If you were to perform a typical black box functional test, you would follow these steps in the GUI of the SUT:
- select a black and white image from your computer
- launch the command that analyzes the image
- check to ensure the output states that the image is black and white
Our software has a GUI. The natural way to do a functional test is to launch computations via this GUI. However, such a computational software must also have a Command Line Interface (CLI). The GUI is the “human-friendly” way to use the software, but the CLI would be useful to launch the computation in the form of “batch processing.” Our advice would be to use this CLI as the main entry point for your testing. This could seem “not so black-box” initially because we are not testing the product as a whole. However, if you are confident enough that the real value that you want to test lies in the engine, then you should test below the GUI. This is what Martin Fowler calls subcutaneous tests.
How many tests?
How many different input images should be tested? Well, as many as you need in order to cover all the functionalities of our software properly. All of the business requirements should be tested. Besides, you should add edge cases that were not specified but may happen. For a black and white feature, our input may vary on many parameters:
- Color: black and white, grayscale, color
- Size: resolution, file size
- File format: JPG, PNG, GIF etc.
Each tested feature requires a specific suite of tests using specific images. In addition, any bug raised internally, or by a customer, should be reproduced with a functional test and added to the set of existing tests. This means that you will likely end up with many tests. Each test might take a significant amount of time to execute. By this, the overall testing time might be hours rather than minutes. This is the price you pay for a well-tested product.
Do you really need to test that often?
Once you have a set of functional tests, the next question is about how often you should perform them. With Agile methodologies and devops movement, the whole software industry is expecting shorter development cycles. Maybe your product is only delivered once a year and such continuous testing seems superfluous. In fact, even if you do not deliver your product often, you should build it an iterative and incremental way. Each new increment should be functionally tested to check that all of your functional tests are still passing. This is essentially regression testing.
Note that a functional test failure does not necessarily mean that there is a functional regression in the SUT. The failure could be caused by a change in the business expectation, a change in the input format, or the command. Even the expected output may vary (typical case: your expectations were, in fact, wrong or approximate and need to be updated). So performing the tests as often as possible will allow you to track (and fix) those changes very quickly.
Need for automation
This quick exploration of functional testing shows that you will need to execute many tests often and each of these tests may have a long duration. As you can see, this won’t be achievable without automation. This is what we will look at in our next article, but feel free to leave comments about what we covered here.
If you are already looking for a functional test automation solution, check-out what we propose and contact us to explore how it could fit your needs. And to stay updated with our product, follow us on Twitter and LinkedIn.
- SUT: System/Software Under Test refers to a system that is being tested for correct operation.
- Subcutaneous test: A test that operates just under the GUI of an application.