Peopleware: Productive Projects and Teams, a book by Tom DeMarco and Timothy Lister about sociology in software engineering, uses the phrase "cult of quality" to define a team that has decided "only perfect is good enough for us." DeMarco and Lister write that while most of the world won't argue for higher quality, members of a cult of quality will "always turn out something that's better than what their market is asking for." And this usually doesn't happen because of any dictate from upper management. Most often it's the team members themselves holding themselves to a higher standard.
I'm senior software developer and squad leader at defi SOLUTIONS, a Software as a Service platform that offers a number of services to the lending industry, and I’m going to take you behind-the-scenes at defi and give you a glimpse into our cult of quality. It's probably easy to understand why an engineering team would want to become a cult of quality; the challenge is how to make that goal a reality. How do we ensure our products meet our own lofty quality standards?
Our clients tell us that one of the most important attributes they want out of a system is stability, yet new features are important, too. These two desires are inherently at odds with one another. New features, even ones that are added smoothly and without any problems, are changes, and therefore are disruptions to the norm.
So how do we serve both masters? The key to introducing new features while keeping the system stable is testing. Thorough testing of a release candidate version of our software will find inadvertently-added bugs as well as verify all pre-existing functionality continues to operate as designed.
Our first line of defense is unit testing, a software engineering technique that tests the components that make up a system at the smallest possible level. For example, applying the principles of software unit testing to a door's deadbolt lock would generate test cases like:
- Does the thumbturn move from side to side?
- Does the bolt move back and forth?
- Is the bolt the desired length?
- Is the box the desired depth?
Note that at this stage, we test all components in isolation. We test the bolt independently of the thumbturn, and we test the thumbturn independently of the box. The point of this stage of testing is to determine if the individual components meet their specific design goals when free of influence from any other component.
But a single bolt doesn’t secure a door, and a single software component doesn’t provide lenders with freedom. A cult of quality cannot limit itself to unit testing. We also must confirm our components work together as expected. In software engineering terms, this is called integration testing.
Integration test scenarios for a deadbolt lock might include:
- When someone moves the thumbturn, does the bolt move outward?
- Does the bolt fit properly into the box?
This video demonstrates a door lock that might have been properly unit tested, but it was not integration tested at all.
User interface tests
Our component has been thoroughly unit tested to ensure it meets all its design requirements. It has been tested in conjunction with other components to make sure they all play nice together. Are we done? Unit and integration testing might be rigorous enough for some organizations, but not for a cult of quality. At defi, our testing isn’t complete without user interface tests.
User interface tests, sometimes called regression tests, enact lengthy scenarios to verify that real world workflows operate as intended. A user interface test for our deadbolt lock would likely be a single test with numerous steps:
- When the lock is engaged, is the door unable to be opened?
- Does the key fit into the lock?
- Does the key turn the lock?
- When the lock is released, can the door be opened?
- Can the key be successfully removed from the lock?
- With the door closed, can the lock be re-engaged from the inside via the thumbturn?
These tests are sometimes called regression tests because once the scenario is written, it can be run over and over, anytime the software changes. When used in this manner, the test is looking for a regression -- a feature that worked once but has been broken.
Unit, integration, and user interface tests vary in the scope of their testing. Unit tests verify behavior at the smallest building blocks of a system, while user interface tests verify complex scenarios function just as they’re supposed to. But a cult of quality holds each type of testing to the same high standards of:
- No human intervention should be needed to run our tests. Every software change should automatically be subjected to each kind of testing, and should only be released once all tests pass.
- Repeatability. Every test must set up its scenario, enact it, verify expected behavior, and clean up after itself over and over and over.
- Relevant. We could write a test that verifies the system acts as expected when a user enters “banana” into each and every text box on the screen, but is that something a real-life user is likely to do?
- Speed. We can’t sit around and wait two days to know whether or not our change works as expected. All of our tests need to run in a reasonable amount of time, even if we have a lot of them. (And we do.)
Every release of the defi system is tested at multiple levels, from individual components all the way up to everyday scenarios. We ensure each type of testing is automatic, repeatable, relevant, and fast. And we’re constantly looking for ways to improve.
All of this means we’re perfect, right? We never, ever issue a new release of our software that contains a bug? Well, no. With a system as feature-filled and flexible as the defi system, there’s a lot of ground to test, and even we make mistakes.
But to us, mistakes simply put the focus on new challenges. What can we learn from it? How can we not repeat it? Can we add enough tests that bugs bounce off of our software like bullets off of Superman’s chest? Yes, we can – and the defi cult of quality expects nothing less of ourselves.
* * *
Michael Ripplinger, a senior software developer with defi SOLUTIONS, has been in the software engineering industry for more than15 years as a developer, team lead, and architect. Mike calls software one of the most powerful tools ever invented and believes its ability to transform businesses and lives barely been exploited.
Also read from Mike Ripplinger: