Rate Limiting Your Workers to Play Nicer with Other API’s
Overview
Many third-party API's have rate limits to ensure you don't overuse or DoS them. Twitter for instance has the following rate limits:
- Unauthenticated calls are permitted 150 requests per hour. Unauthenticated calls are measured against the public-facing IP of the server or device making the request.
- OAuth calls are permitted 350 requests per hour and are measured against the oauth_token used in the request.
Now that's not a lot of requests! 2.5 requests per minute if unauthenticated. So how can we slow down/throttle our massive queue of Twitter-related tasks in IronWorker so we don't hit that limit?
Using IronWorker's 'delay' parameter for Rate Limiting
IronWorker supports a 'delay' parameter (docs) when you are queuing up a task which allows you to specify a time to wait before the task will be eligible for execution.
So let's use Twitter, for example, if we have 10,000 tasks and each task will hit twitter once, we can skew our tasks to ensure we don't hit the rate limit. If we can do 2.5 requests per minute against the Twitter API, then let's stick to 2 IronWorker tasks per minute (leave a little room just in case). So here's how you could queue up those jobs:
And there you have it. A nice, simple way to throttle your workers.
Full code example is here.
Related Reading: Top 10 uses of ironworker
Unlock the Cloud with Iron.io
Find out how IronWorker and IronMQ can help your application obtain the cloud with fanatical customer support, reliable performance, and competitive pricing.
← Platforms, Languages, and App Services: Economies and Communities of ScaleWhen Unit Tests Aren’t Enough… ABT – Always Be Testing →
This also works really well when scheduling tasks:
start_at = Time.now
10000.times do |i|
worker = MyWorker.new
worker.schedule start_at: start_at
start_at += 30
end
Hi Luke,
We recommend to use the ‘delay’ parameter on queue rather than using the scheduler to do these kinds of things because there is a limit on the number of schedules you can have and it’s a lot more efficient for us on the backend.
Question on pricing… Does the delayed time count as machine time?
Hi Jared,
No it doesn’t count towards machine time. The delay time is free.