Automated Quality Gateways

Maarten Groeneweg

In the medieval times of software development we kept bugs outside by building a big wall between development and production. We mostly called it the Quality Assurance department. They acted as gatekeepers, every piece of functionality was closely examined and if it contained bugs it was rejected at the gate.

That worked out quite well, but the world changed. Nowadays there is a need for bringing stuff to production fast. So agile methodologies and Continuous Delivery are hot and we got rid of our gatekeepers. Don’t get me wrong, I really like the agile movement but it seems that we forgot to implement a new way of validating quality.

Exile the monolith

The old way of Quality Assurance was basically a monolithic approach. There was one big system/department in place which had many responsibilities. Moving away from that allows us to move faster and respond better to the rapidly changing playground. But I don’t believe we should ditch our QA process. Fast delivery is nice but if the quality ain’t good you won’t end up with happy users.

So there is a need for a faster and more distributed system to incorporate Quality Assurance within our agile development process. The solution: Automated Quality Gateways.

What are Automated Quality Gateways?

See Automated Quality Gateways as micro service infrastructure for your quality system. Every single gateway has one thing which it can do very well (and fast). They are focused on finding a specific kind of defects in your application on a specific moment in your development process.

A good Automated Quality Gateway should always comply to five rules. If not then it’s not an Automated Quality Gateway.

1. Triggered automatically

No need to press buttons, it’s triggered automatically based on a event. Which event? That depends on at which stage in the development cycle it can run. Typical events are: a commit, push, merge or deploy.

2. Defects are blocking the process

If the gateway finds a defect this should block the process. So it should not be possible to continue with the next step of the development cycle. By example, it shouldn’t be possible to merge a branch when it still has failing unit test.

3. Quick feedback

Slow feedback is useless, especially when it’s blocking your process. So Automated Quality Gateways should be as fast as possible. This allows you to implement them in an early stage of your development cycle. If it only takes 1 second to run then it can be done after each commit. But if it takes minutes you’ll be forced to implement it closer to the end of the cycle.

4. Mitigate a specific risk

To prevent that a gateway will become a cute little monolith with all the downsides like slower execution time and inflexibility you should make sure that they are focusing on mitigating a specific risk. So don’t check for visual defects in your integration test suite and don’t do integration testing in your unit test suite. Sounds easy...but I saw it happening way to often.

5. You can trust them

If a piece of code passed the gateway than the teams should have trust in it. You don’t want to manually check the same things again. On the other hand, you don’t want to have any false positive either. In my experience there is nothing so destructive for a test suite than random failures. People will lose their faith in it and start disabling tests and ignoring failures.

Automated Quality Gateway System

One gateway does not necessarily provides a lot value. But when you build a system of multiple gateways then it will become very powerful. In the diagram below you will find an basic example of an Automated Quality Gateways System which is mapped on the Git Flow process.

  • After each commit on a feature branch it does a compile

  • After each compile it runs the unit tests

  • Before a merge to DEV it runs the API tests

  • After the API tests it runs a code quality tests and validates it doesn’t violate any rules

  • After the code quality tests it will run the integration test

  • Before Dev is merged into master the security tests are ran


What’s a good Automated Quality Gateway?

The most important thing is that it adds value. It may sounds awesome to have thousands of end-to-end tests but if you spend 50% of the development budget on them and you’re not finding defects it’s bummer.

It is important to take the costs of a gateway into account. They can be very expensive to build and maintain. In general it’s cheaper to cover risks on a lower level. By example, unit tests are cheaper than end-to-end test. So it’s smart to cover as much as possible on that level. Don’t trow away all end-to-end tests but let them focus on the integration of all components instead of trying to cover every scenario.

On the other hand each gateway should add value. So first of all they should actually help you find defects. But the teams also need to trust the total quality system. It makes no sense to invest a lot of time and you’re still want to do you’re manual regression tests. Finding a bug late in the process is expensive so implementing a gateway which finds defects early can add a lot of value.

How to use it?

The best way to implement a systems of Automatic Quality Gateways is incremental. Simply start implementing your first Automatic Quality Gateway. A good start is to ensure that your unit tests are ran after each commit. does the first gateway run stable and does it provide sufficient coverage? Then start implement the next gateway and so on.


Automation Testing Validation Continuous Deployment