When Unit Tests Aren’t Enough… ABT – Always Be Testing
Traditionally, developers create unit tests to test code and to ensure bugs don’t get introduced as time goes on and your code changes. The theory is that if you write tests that cover every line of code, then when you make a change and the tests still pass, you can feel pretty comfortable that it will work as it should.
Well there is a lot more that can go wrong in your production system than introducing a bug in your code.
What Else Can Go Wrong?
Your database could fail, your hardware could fail, your cloud provider could fail, your network could fail, your load balancer could fail, your users could do something unexpected causing a failure, etc, etc, etc. There are so many points of failure in a full production system that testing your code alone just isn’t enough, not even close.
Always Be Testing
One of Iron.io‘s core values is reliability. All of our services are built with reliability in mind meaning we do whatever we can to ensure that our services are running at all times and that our users can rely on us when building their applications. We provide core infrastructure services and our users depend on us to be reliable because if we go down, they go down. Continuously testing our services is part of ensuring that we are reliable and we are always looking for new ways to test and monitor our systems so we find problems before any of our users do.
One of the best ways to test every piece of a system is to test the piece that is closest to the user whether it’s the user interface or an API. Testing at the edge ensures that we’re testing the entire system including the API/UI, the servers, the databases, the hardware, third party services, etc. We’re testing from the top all the way to the bottom of the stack and back again.
The ABT (Always Be Testing) Gem
I created a simple testing framework (for Ruby) called abt that given a git repository URL, will clone the repository, run the tests and send out notifications (customizable). This is not the same as Continuous Integration (CI) as you know it. A typical CI server will kick off your unit tests when someone commits changes to code. ABT is run on a schedule using IronWorker so the tests are run regardless if the code has changed.
And it’s easy to use, here’s a step by step.
1) Get an Iron.io account so you can use IronWorker
If you don’t already have an account, go sign up now at ironio.staging.wpengine.com, it’s free.
2) Install abt gem
sudo gem install abt
3) Ensure the library you want to test can use the ABT config parameters
The tests should check for a global $abt_config variable, for example:
Have a look at this file for a real world example in the iron_mq gem.
4) Queue it up on IronWorker to run the tests
There is some configuration for IronWorker and configuration for the test suite you want to run. Then we just specify a git URL and pass along the configuration in test_config. Finally, we add some notifiers, in this case we’re using the included HipChatNotifier, but you can create your own to do whatever you want whether it’s sending an email, notifying PagerDuty or posting a Tweet.
So after it runs we’ll get notified in our HipChat room if it passed or failed with the details.
Here is a post to HipChat for a failing test:
5) Schedule it to run continously!
Now let’s schedule it to make it run every 30 minutes. Just change worker.queue in the code above to:
This will run the first test right away and then repeat the test every 30 minutes indefinitely.
Conclusion
So there you have it. A simple way to constantly be testing your production and/or staging environments. Constant Integration testing if you will. And it’s pretty easy to make any ruby client library work with the abt gem (it just needs to know how to access the configuration), so you can start using it right away.
If you are running a service people depend on, you should abt, always be testing.
Source Code and more Information: https://github.com/iron-io/abt
