Best practices #1: Test quarantine

Many companies have a lot of regression tests but quite often are struggle with bunch of them failing every build or from time to time (flaky tests). Sometimes it is 25% or more of all of automation tests. What should you do as „quality” engineer in this situation? Should you stop deployment for months until you and your team to fix all of unstable and failing tests or should you manually check every single failure before every release to production? Or maybe you should just ignore it and wait till the situation will improve somehow itself?

Well, there could be a lot of reasons of failing tests. Some of them can be outdated and require small update. Some may have wrong design or be unstable because of buggy test code. Probably your test environment is not perfect too. But still some of these failures may be correct and there is a bug in the system.

In most cases stopping deployment process for weeks or months is not an option. Management wouldn’t approve this strategy. They need to release new features and make the system to grow. Manually checking everything every time you go to production is just wasting time. Once you finish you need to start doing it again and so on. And as you probably can guess ignoring this situation is not good either. So what to do?

The solution is quite simple and its name is „Test quarantine”. The idea is to mark all failing tests as „unstable” and do not run them with the test suite. You just need to exclude them from execution. This way only „green” tests will stay. And the good news is you can trust them from now. If your tests are red there is a bug (either in your test or application code). And from now on you need to remember that you can’t deploy the code to production when „green” suite is failing. Simple as that.

And what about all of these tests marked as „unstable”? The good idea is to introduce a rule of fixing one test from „unstable” suite every week. If every team or every developer would follow this rule you would clean your tests faster then you can imagine 🙂

And last thing. Test quarantine is one time action. You should not let anyone to extend „unstable” suite. Adding tests to quarantine is forbidden after „green” suite were selected. If any test from „green” suite is failing it needs to be fixed. No exceptions from that rule.

Fingers crossed for your test suites 🙂 Hope this article help you make your deployment process better 🙂

5 Replies to “Best practices #1: Test quarantine”

  1. I have acquired some new items from your web site about computer systems. Another thing I have always presumed is that computer systems have become something that each family must have for several reasons. They supply you with convenient ways in which to organize homes, pay bills, shop, study, listen to music and perhaps watch shows. An innovative approach to complete all of these tasks is by using a notebook. These desktops are portable ones, small, strong and mobile.

  2. Most of the things you articulate is astonishingly legitimate and that makes me ponder the reason why I hadn’t looked at this with this light previously. Your piece truly did turn the light on for me as far as this issue goes. Nonetheless there is actually one particular issue I am not really too comfortable with and while I attempt to reconcile that with the main theme of your position, let me observe just what all the rest of the subscribers have to point out.Nicely done.

  3. Wonderful work! This is the type of info that should be shared around the web. Shame on the search engines for not positioning this post higher! Come on over and visit my web site . Thanks =)

  4. Does your site have a contact page? I’m having a tough time locating it but, I’d like to shoot you an email. I’ve got some creative ideas for your blog you might be interested in hearing. Either way, great website and I look forward to seeing it grow over time.

  5. Have you ever thought about creating an ebook or guest authoring on other blogs? I have a blog based upon on the same information you discuss and would love to have you share some stories/information. I know my viewers would enjoy your work. If you are even remotely interested, feel free to send me an email.

Comments are closed.