Event Handling with .NET, RaventDB, and IronMQ

The other day we blogged about a map-reduce contribution from the Iron.io community. Here’s another great contribution from an Iron.io user which highlights handling commit transactions within a database all the while coordinating downstream events using a message queue. 

Jef Claes is a developer in Belgium that has written a few things about IronMQ in the past (here and here). His latest blog post addresses maintaining eventual consistent domain events using RavenDB and IronMQ. It’s a short post but a good rundown on how to maintain consistency when transactions involved multiple distributed systems.

About the Technologies

RavenDB is a transactional, open-source Document Database written in .NET, offering a flexible data model designed to address requirements coming from real-world systems.

IronMQ is a scalable cloud-based message queuing service, designed for building distributed cloud applications quickly and operating at scale. It provides and easy mechanism to control work dispatch, load buffering, synchronicity, database offloading, and many other core needs.

–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––

One project I'm working on right now makes use of domain events. As an example, I'll use the usual suspect: the BookingConfirmed event. When a booking has been confirmed, I want to notify my customer by sending him an email.

I want to avoid that persisting a booking fails because an eventhandler throws - the mail server is unavailable. I also don't want that an eventhandler executes an operation that can't be rolled back - sending out an email - without first making sure the booking was persisted successfully. If an eventhandler fails, I want to give it the opportunity to fix what's wrong and retry.

Get In Line

Code Example from the Post

The idea is, instead of dealing with the domain events in memory, to push them out to a queue so that eventhandlers can deal with them asynchronously. If we're trusting IronMQ with our queues, we get in trouble guaranteeing that the events aren't sent out unless the booking is persisted successfully; you can't make IronMQ enlist in a transaction.

Avoiding False Events
To avoid pushing out events, and alerting our customer, without having successfully persisted the booking, I want to commit my events in the same transaction. Since IronMQ can't be enlisted in a transaction, we have to take a detour; instead of publishing the event directly, we're going to persist it as a RavenDB document. This guarantees the event is committed in the same transaction as the booking.

Getting the Events Out
Now we still need to get the events out of RavenDB. Looking into this, I found this to be a very good use of the Changes API. Using the Changes API, you can subscribe to all changes made to a certain document. If you're familiar with relation databases, the Changes API might remind you of triggers - except for that the Changes API doesn't live in the database, nor does it run in the same transaction. In this scenario, I use it to listen for changes to the domain events collection. On every change, I'll load the document, push the content out to IronMQ, and mark it as published.

A Back-up Plan
If the subscriber goes down, events won't be pushed out, so you need to have a back-up plan. I planned for missing events by scheduling a Quartz job that periodically queries for old unpublished domain events and publishes them.

In Conclusion
You don't need expensive infrastructure or a framework to enable handling domain events in an eventual consistent fashion. Using RavenDB as an event store, the Changes API as an event listener, and IronMQ for queuing, we landed a rather light-weight solution. It won't scale endlessly, but it doesn't have to either.

–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––

More on Patterns in Distributed Processing

The post does a great job of highlighting issues that arise building transaction-based systems using distributed cloud technologies. While you get tremendous ease of use and agility, developers do need to take into account the sequence and order of committing events and the ability to make sure that downstream events are only processed after the primary events have been recorded.

Message queues can be essential components for this but any production-ready solution also means having checks in place along with solid exception handling and good back-up plans. Stay tuned for more posts on distributed processing patterns in the future.

Notes on the Post

• The latency hit mentioned in the article can be almost all attributed to the cross-atlantic message transit. Doing the processing in the same or nearby datacenters as the message queue will cause message transit to go into single ms territory. IronMQ is currently in AWS-East and Rackspace-ORD (and coming to a datacenter near you). 
• And as for scaling the solution, the gating factor is more with database inputs than with IronMQ. IronMQ is designed to handle very large throughputs and so it should handle whatever is thrown at it.