Monitor IronWorker Tasks with Prometheus!

Monitoring IronWorker Metrics with Prometheus

IronWorker task’s workflow

When you push a new task to IronWorker, it puts it into the queue and sets its state to “queued”. The runner is the “agent” that permanently polls this queue, retrieves the task, and runs it. When the runner starts processing the task - it changes the status to “preparing”. At this stage, the runner pulls the docker image and other stuff it needs to run the task. After the preparation step is done - the state is changed from “preparing” to “running” and the runner starts running the code of this task. Depending on the result, the final status can be:

• “complete” (task completed successfully)
• “error” (the task failed, due to an exception in the code for example)
• “timeout” (task’s duration exceeded the timeout limit)
• “canceled” (user manually canceled the task before it stopped)

IronWorker endpoint for Prometheus

IronWorker has an API endpoint that returns the data in Prometheus exposition format:

https://worker-aws-us-east-1.iron.io/2/projects/YOUR_PROJECT_ID/codes/YOUR_CODE_ID/metrics

In the above endpoint, YOUR_PROJECT_ID and YOUR_CODE_ID should be replaced with actual values. It returns the number of tasks in each of seven possible task statuses for the specified worker (by your code_id): queued, running, preparing, error, timeout, canceled, complete.

Configure Prometheus to run locally

Running Prometheus in Docker is very simple, especially for people who are familiar with Docker. First, we need to add the IronWorker endpoint to its configuration file (prometheus.yml):

global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
rule_files:

# Here it's Prometheus itself.
scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["worker-aws-us-east-1.iron.io"]
    metrics_path: "/2/projects/YOUR_PROJECT_ID/codes/YOUR_CODE_ID/metrics"
    params:
      oauth: ['YOUR_TOKEN']

Now we are ready to launch the Prometheus service locally and mount the above file to a docker container:

docker run --rm -it -p 9090:9090 -v /path/to/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus

Navigate to localhost:9090 URL in your browser. You will be able to access the following metrics:

• worker_tasks_queued
• worker_tasks_running
• worker_tasks_cancelled
• worker_tasks_complete
• worker_tasks_error
• worker_tasks_preparing
• worker_tasks_timeout

Two custom labels come with the above metrics: project_id, code_id, i.e. it's possible to filter by them.

You can configure the alerts depending on the number of failed IronWorker tasks, i.e. if the number of failed tasks reaches a threshold value - the alert can be triggered. Also, you can configure an alert when you’re experiencing processing delay, during peak times for example. This can be achieved by specifying a threshold value for “worker_tasks_queued” metric.

Finally, you may use tools like Grafana to visualize the metrics collected by Prometheus for a comprehensive view of your task performance.