Treatise on Testing Modern Software

home | photography | software engineering | talks | About

It was a bright sunny morning when I read the unbelievable news - the Mars Rover was destroyed because the engineers who built it did not convert metric to English system and tragedy ensued. My younger self at that time could not fathom the idea that a simple conversion could bring an untimely end to the 125Million dollar project! After working for 5 1/2 years as a Software Test Engineer and Developer for Office 365 at Microsoft, my experience has been that the probability of such disasters could be used minimized by testing.

Newspaper cartoon depicting the Mars orbiter disaster (source: slideplayer.com)

As the complexity of the software projects grow, so does the difficulty of testing those big software projects. With the growth of cloud technologies, the number of users who use a software now reaches hundreds of millions and even billions. When mistakes are made, the cost of that mistake can now potentially be catastrophic. I often think about the role of testing and decided to write a birds eye view of the testing process in the software industry - the tale from trenches so to speak.

Unit Test

Unit testing is the first line of defense against those creepy crawlies Once I did not add enough unit tests in my code, a senior developer asked me "How do you know that your code works?" When practicing TDD (test driven development), you would write a test for your new piece of code, run the test which will fail, and then write code until that failing test passes. The unit tests are usually white-box tests - tests that know how a unit/function operates. While building a car, a unit test would be measuring and verifying the current produced by the alternator.

Functional tests

The functional tests takes on the dependencies of external libraries and services to test the functionality of an entire computer program or module. While building a car, functional testing would be measuring the torque while running the engine with the input of gas and battery. Tools such as Sonarqube can help you run all unit and functional tests on demand or as part of the CI/CD pipeline.

Integration Tests

These are the heavyweights of the testing world. These are designed to test the software end-to-end with in a real-world environment. You can think of these as driving the automobile on a road to test how well it drives. In this test all components of the car are working in unison to produce the driving outcome. Integration tests are usually the most time-consuming tests to write and are the most difficult to maintain.

Performance Tests

So, the car drives fine in a plane obstacle-free road, but what about when the going gets tough? Performance tests are all about testing the program as a whole under load conditions. You can think of this as driving the car in with maximum load (passengers or goods). Performance tests are usually done after completing all other testing has been done and the infrastructure and code have reached a certain level of stability. Tools such as Jmeter are useful for running performance/load tests on APIs.

Monitoring

In cloud computing, the major components/modules of a piece of software is stored in the cloud and is updated frequently by a moderately large group of engineers. To ensure the QoS, the critical components of a software are now monitored continuously. The monitor is usually a program that measures the health of some service or executes a set of defined steps over and over. When the health metric of the service falls below threshold or one of those steps fail, the monitor alerts the supporting engineering team. In our car example, it would be like checking the battery power and the engine temperature while the car is being driven. Finding the right balance between effective monitoring and alerting noise will take continual testing and re-adjustment of alert threshold.

Optics

While monitors detect instantaneous failures issues, optics are a way of looking at the performance and history of the software in the real world over the long term. Optics for a software may provide insights into its usage (growth of user base), performance (response time during peak and off-peak hours), etc. For our car example, optics would be like collecting and analyzing gas mileage data for the car over a month to verify its actual mileage. A successful optics infrastructure should contain a small set of basic reports and the ability to export collected data. Periodic analysis of the exceptions and performance data collected from your production environvent is an effective method of improving your quality of service (QoS).

Program testing can be used to show the presence of bugs, but never to show their absence!