Many companies have a lot of regression tests but quite often are struggle with bunch of them failing every build or from time to time (flaky tests). Sometimes it is 25% or more of all of automation tests. What should you do as „quality” engineer in this situation? Should you stop deployment for months until you and your team to fix all of unstable and failing tests or should you manually check every single failure before every release to production? Or maybe you should just ignore it and wait till the situation will improve somehow itself?
Well, there could be a lot of reasons of failing tests. Some of them can be outdated and require small update. Some may have wrong design or be unstable because of buggy test code. Probably your test environment is not perfect too. But still some of these failures may be correct and there is a bug in the system.
In most cases stopping deployment process for weeks or months is not an option. Management wouldn’t approve this strategy. They need to release new features and make the system to grow. Manually checking everything every time you go to production is just wasting time. Once you finish you need to start doing it again and so on. And as you probably can guess ignoring this situation is not good either. So what to do?
The solution is quite simple and its name is „Test quarantine”. The idea is to mark all failing tests as „unstable” and do not run them with the test suite. You just need to exclude them from execution. This way only „green” tests will stay. And the good news is you can trust them from now. If your tests are red there is a bug (either in your test or application code). And from now on you need to remember that you can’t deploy the code to production when „green” suite is failing. Simple as that.
And what about all of these tests marked as „unstable”? The good idea is to introduce a rule of fixing one test from „unstable” suite every week. If every team or every developer would follow this rule you would clean your tests faster then you can imagine 🙂
And last thing. Test quarantine is one time action. You should not let anyone to extend „unstable” suite. Adding tests to quarantine is forbidden after „green” suite were selected. If any test from „green” suite is failing it needs to be fixed. No exceptions from that rule.
Fingers crossed for your test suites 🙂 Hope this article help you make your deployment process better 🙂