Building an Analytics Engine using MongoDB, Go, and Iron.io
 |
| Building a Relevancy Engine Using MongoDB and Go |
On the heels of a recent post on powering intelligent traffic systems using MongoDB and Iron.io comes a presentation on building an analytics engine using MongoDB, Go, and Iron.io.
William Kennedy gave a presentation on his recent work at GopherCon and friends of ours from Sourcegraph were kind enough to write up details on the talk.
Here's an excerpt from their summary.
The search for a solution
The first version that Bill built used a SQL database. But whenever they wanted to make a new feed, they’d have to build a new table, populate it with data, etc. That would take a long time. This solution wasn’t working out for them.
They needed a system that could easily and quickly create dynamic feeds based on all of the data. They also wanted to be able to write rules to alter the overall content of a user’s feed quickly (for example, to create custom Valentine’s Day suggestions).
Their system needed to allow them to:
• Write rules that can be updated and applied at runtime
• Pass variables to filter and pinpoint relevance
• Use data aggregation technique to filter and group data
• Build tests around aggregated datasets
• Build tests against multiple aggregated datasets
• Publish data from offer/deal feed and other internal feeds
After evaluating a number of other tools, they settled on Go, Linux, MongoDB, beego, mgo, and Iron.io. (They choose beego over other Go Web frameworks because they liked its MVC architecture.)
They used a denormalized schema for their feed data and kept it updated using workers running tasks on Iron.io.
Q&A
Q: How well do you think this will scale? A: I went with MongoDB because of its scalability. It is scaling much better than the previous SQL database solution. Iron.io is also super helpful in allowing this system to scale easily.
To read the full summary, go here.
Here's the video:
––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
William writes Going Go Programming and is one of the authors of Go in Action and so he knows what he's talking about when it comes to using Go. His posts are well written and have great technical details and so we highly recommend spending time with his posts.