Comments on: How to Reduce a 9 Hour Job into a 9 Minute Job https://blog.iron.io/how-to-reduce-9-hour-job-into-10-minute/ Scalable serverless application tools to run background tasks with Docker containers and manage messaging queues with cloud elasticity and handheld customer support. Wed, 02 Aug 2023 04:58:36 +0000 hourly 1 https://wordpress.org/?v=6.9.4 By: Travis Reeder https://blog.iron.io/how-to-reduce-9-hour-job-into-10-minute/#comment-41 Tue, 01 Jan 2013 21:31:38 +0000 http://ironio2.wpengine.com/?p=201#comment-41 Hi Unknown, worker hours are based on $0.05 per hour, not $0.75 (that’s the overage price if you exceed your plan limits) and the actual running time of each task was ~6 minutes (“Each task should take approximately 3.5 * 100 = 350 seconds = ~6 minutes to run”) so it would be:

2000 * 6 / 60 * $0.05 = $10

]]> By: Unknown https://blog.iron.io/how-to-reduce-9-hour-job-into-10-minute/#comment-40 Tue, 01 Jan 2013 18:49:36 +0000 http://ironio2.wpengine.com/?p=201#comment-40 Wait a minute, that is 8.75 minute per worker process. So if you have 2000 worker process in parallel, ironworker will charge you for 8.75*2000*0.075/60 = $21.82

true?

]]> By: Trevor https://blog.iron.io/how-to-reduce-9-hour-job-into-10-minute/#comment-39 Tue, 04 Dec 2012 16:14:37 +0000 http://ironio2.wpengine.com/?p=201#comment-39 You could use `find_in_batches`and avoid your custom batching:
https://apidock.com/rails/ActiveRecord/Batches/find_in_batches

]]> By: Travis Reeder https://blog.iron.io/how-to-reduce-9-hour-job-into-10-minute/#comment-38 Mon, 12 Nov 2012 22:05:19 +0000 http://ironio2.wpengine.com/?p=201#comment-38 We recommend a run-time of at least 30 seconds for each task so that would be a good starting point. Really, the more you can do in a single task, the more efficient it is because you amortize the setup/teardown time of your worker (loading, making db connections, etc). But at the same time, there is value in having short quick tasks, for instance if task errors out, it’s easier to debug and retry it. Also, you don’t have to wait forever to ensure that it worked (or didn’t).

So as a rule of thumb, greater than 30 seconds, but less than 5 minutes is a good place to be.

]]> By: IPDb Developers https://blog.iron.io/how-to-reduce-9-hour-job-into-10-minute/#comment-37 Sat, 10 Nov 2012 20:23:51 +0000 http://ironio2.wpengine.com/?p=201#comment-37 Can you provide some insight as to how to choose batch size for each IronWorker? That is – if my task has a set up time of X and a run time of Y, how many tasks should I send to each IronWorker?

]]> By: Travis Reeder https://blog.iron.io/how-to-reduce-9-hour-job-into-10-minute/#comment-36 Mon, 04 Jun 2012 21:32:25 +0000 http://ironio2.wpengine.com/?p=201#comment-36 Hi Sunny,

100 users in the example above is arbitrary. It’s your code and your choice to do it however you want, you could do 1 user per worker or 1000 per task if you wanted.

The final task would be whatever is left over. If you had 200,001 users, then the final task would just do the work for 1 user. Again, it’s your code and you create the payload for each task so it would be totally up to you.

]]> By: Sunny Gleason https://blog.iron.io/how-to-reduce-9-hour-job-into-10-minute/#comment-35 Mon, 04 Jun 2012 21:09:34 +0000 http://ironio2.wpengine.com/?p=201#comment-35 Are there always a multiple of 100 users? If not, it seems like this code might be missing the part to enqueue the final tail remainder…

]]> By: Travis Reeder https://blog.iron.io/how-to-reduce-9-hour-job-into-10-minute/#comment-34 Wed, 02 May 2012 17:27:48 +0000 http://ironio2.wpengine.com/?p=201#comment-34 Hi Ismael, could you share your calculation?

While it’s running, the cost would be the same, but with IronWorker, you would only pay for the time the workers are actually running, no more, no less. With Heroku you’d have to pay for the 9 hours of time and then be sure to turn off all your workers every day after they were done or pay $827 per month to keep them running.

IronWorker would be ~$10 per nightly run (2000 * 6 / 60 * $0.05)

]]> By: ismael https://blog.iron.io/how-to-reduce-9-hour-job-into-10-minute/#comment-33 Mon, 26 Mar 2012 21:24:29 +0000 http://ironio2.wpengine.com/?p=201#comment-33 In my calculations this way saves them about $340 in workers, is it true?

]]>