Subha Navaratri!

Background job processing in Rails with delayed_job

Note

This blog post describes how to use delayed_job with Rails 2. For Rails 3, see this instead.

Also, in case you are facing problems with daemons gem, you can use daemon-spawn as an alternative as described in this post or directly go here

Why background processing is needed

Since Ruby 1.8 only supports green threads, if you want to process something longish as part of a web request in Rails, it will delay the response. Of course, if the processing is not critical to returning a response to the client, it’s wiser to push the processing to a background queue and return a response immediately.

There are other incidental benefits to using a background job processing queue like, letting a dedicated server, other than the customer facing app server, handle it, assigning priority to jobs, retry on failure etc.

Most common situations where such a background processing is needed are: sending emails to users, rebuilding search indices, image and video processing, etc.

Available options

There are several ways to process background jobs in Rails. In fact, So many that Geoffrey Grosenbach joked on the RailsEnvy podcast about the number of continuous integration servers finally being equal to number of queuing servers available in Ruby.

My favorite one so far is: delayed_job. Because its simple to set up, simple to use and has lot of powerful features. However, don’t forget to look at others to see what fits you needs and style better. For an overview see here. In the rest of the article, I will explain how to install and use delayed_job.

Installation

Firstly, install it in your rails project just as you would any other plugin:


$ ruby script/plugin install git://github.com/tobi/delayed_job.git

Delayed_job stores the jobs in a table in the database. Create the table using
a migration:


$ ruby script/generate migration create_table_for_delayed_job

class CreateTableForDelayedJob < ActiveRecord::Migration
  def self.up
    create_table :delayed_jobs, :force => true do |t|
      t.integer  :priority, :default => 0
      t.integer  :attempts, :default => 0
      t.text     :handler
      t.text     :last_error
      t.datetime :run_at
      t.datetime :locked_at
      t.datetime :failed_at
      t.string   :locked_by
      t.timestamps
    end
  end

  def self.down
    drop_table :delayed_jobs
  end
end

$ rake db:migrate

Note: Compared to the documentation, I have changed last_error column to text from string to handle larger error messages

Delay jobs

Restart your server, and you can start delaying jobs. There are two ways to push a job to background. First, the easy way:

You have seen Ruby’s send method, right.


"Bangalore".length # returns 9
"Bangalore".send(:length) # same as above, returns 9

send just invokes the given method on the object.

Analogous to that, delayed_job adds send_later method. Say, if you were sending a mail to user on registration:


NotificationMailer.deliver_welcome_user(@user)

Just change that to:


NotificationMailer.send_later(:deliver_welcome_user, @user)

And you are done. delayed_job will push this into the queue (the database table you created above), to be executed later.

There is another way to background a job. First, create a job class.


# put this in lib/delayed_notification.rb
class DelayedNotification
  attr_accessor :user_id
  def initialize(user_id)
    self.user_id = user_id
  end

  def perform
   NotificationMailer.deliver_welcome_user(User.find(@user_id))
  end
end

# Add this where you were sending the mail earlier
Delayed::Job.enqueue DelayedNotification.new(@user.id)

Executing the jobs

Now, there are several ways to run the job which are there in the queue. In development mode, you can just issue the following command in your terminal:


$ rake jobs:work

This will cause the worker to run in a loop. Stop it by pressing Control-C.

On production, you would want the worker to be running all the time. Also, it will be nice to have the ability to stop the worker just before the deployment and start it again once the deployment finishes. It sounds complicated, but its pretty easy to set up.

First, install the daemons gem:


# Add in environment.rb
 config.gem 'daemons'

$ rake gems:install

Then, copy this script in your rails project in file script/delayed_job and give it execute permission. This creates a worker daemon, which when started keeps running as a background process.

Alternately, instead of daemons gem, you can use daemon-spawn gem.

Install the gem on the host where you want the delayed_job daemon to run:


$ sudo gem sources -a http://gems.github.com
$ sudo gem install alexvollmer-daemon-spawn

And then copy this script instead in script/delayed_job. Rest all of the following steps are identical, whichever gem and script/delayed_job you choose.


# start the worker daemon
$ ruby script/delayed_job start

# stop it
$ ruby script/delayed_job stop

You would of course want to start and stop it on your production server using Capistrano. Here’s a recipe to do that (courtesy collectiveidea)


# add this to config/deploy.rb
namespace :delayed_job do
  desc "Start delayed_job process"
  task :start, :roles => :app do
    run "cd #{current_path}; script/delayed_job start #{rails_env}"
  end
  
  desc "Stop delayed_job process"
  task :stop, :roles => :app do
    run "cd #{current_path}; script/delayed_job stop #{rails_env}"
  end

  desc "Restart delayed_job process"
  task :restart, :roles => :app do
    run "cd #{current_path}; script/delayed_job restart #{rails_env}"
  end
end

after "deploy:start", "delayed_job:start"
after "deploy:stop", "delayed_job:stop"
after "deploy:restart", "delayed_job:restart"

And you are done. Just deploy as usual.


$ cap deploy:migrations

Further tips and tricks

  • Get a separate server to process your delayed job so as not to take CPU away from customer facing app server.
  • You can customize some parameters like how many times to retry on failure and prune runaway tasks. See here.
  • Delayed_job allows you to assign priority to jobs.
  • In development mode, if you just want to discard pending jobs instead of running them:

$ rake jobs:clear

  • If you want to control the time at which the job gets executed, you can specify it as an argument to enqueue while delaying the job:

Delayed::Job.enqueue DelayedNotification.new(@user.id), 0, 15.minutes.from_now

Here, 0 denotes the (default) priority of the task. This task will get executed at least 15 minutes later than it would have been executed normally.

References

For more about this wonderful plugin, refer to these resources:

22 Comments Added

Join Discussion
  1. Shai
    Shai Mar 18, 2009 at 10:45 AM
    I found some issues (maybe just for me) with the recipe you posted: /usr/local/lib/ruby/gems/1.8/gems/capistrano-2.5.5/lib/capistrano/configuration/namespaces.rb:188:in `method_missing': undefined local variable or method `rails_env' for # (NameError) Not sure if it can access rails_env. What I did to fix it was pretty simple. I switched the delayed_job script to: require File.dirname(__FILE__) + '/../config/environment' Daemons.run_proc('job_runner') do Delayed::Worker.new.start end I also removed #{rails_env} from the end of each task. Hope that helps anyone who has the same issues.
  2. Amit Mathur
    Amit Mathur Mar 20, 2009 at 12:24 AM
    @shai: Thanks for the comment. However, please note that you will need to pass the Rails environment to script/delayed_job if you are going to run on multiple environments (development and production). So, I would advice against modifying it. Usually, people set rails_env in their Capistrano recipe file. If you don't have it set, and are using only development environment, as a quick fix, you can simple say near the top of your config/deploy.rb: set :rails_env, "development"
  3. Shai
    Shai Apr 03, 2009 at 12:24 AM
    Hi Amit. Thanks for the response - I see what you mean. The only issue I have right now with that is when I deploy (script/delayed_job restart production) nothing happens. When I log in remotely to my machine and run script/delayed_job start production I notice that delayed job starts working, but for development. Any idea what I am doing wrong? Thanks.
  4. Amit Mathur
    Amit Mathur Apr 10, 2009 at 9:37 PM
    @shai: Answers to your questions: (a) script/delayed_job restart production not working: restart will work only if the process is already running, so for the first time you should start it manually (script/delayed_job start production). (b) As I said in my earlier comment, you should set rails_env in the capistrano file appropriately. It is most likely not set to "production" for your production host. If you like, feel free to email me directly if you are still not able to make it work (see contact page)
  5. Mauricio Gomes
    Mauricio Gomes Apr 24, 2009 at 12:54 AM
    Does the script to run delayed::job as a daemon work outside of Capistrano? It works for me when I run script/delayed_job run but not when I try script/delayed_job start production.
  6. Ryan L
    Ryan L Apr 30, 2009 at 4:52 AM
    Hey there, i know is not a real tricky thing, but on production, it can't find the daemons gem. Of course it works fine on dev. no such file to load -- daemons i have config.gem 'daemons', :lib => 'daemons', :version => '~> 1.0.10' unpacked in the vendor/gems folder also, can you add the: after "deploy:start", "delayed_job:start" after "deploy:stop", "delayed_job:stop" after "deploy:restart", "delayed_job:restart" just at the end of the deploy file? I have this currently: after "deploy", "deploy:migrations" after "deploy:migrations", "deploy:cleanup" after "deploy:cleanup", "symlinkify" after "symlinkify","build_asset_packeges" after "build_asset_packeges","reload_passenger"
  7. Amit Mathur
    Amit Mathur Apr 30, 2009 at 1:42 PM
    @Mauricio: Yes, the script will run standalone (outside of Capistrano). Make sure the host where you are trying to run it in production mode, is really your production server - it should have the production DB and delayed_jobs table in that production DB.
  8. Amit Mathur
    Amit Mathur Apr 30, 2009 at 1:43 PM
    @Ryan: Perhaps some problem with your gem set up. BTW, I have updated the article to give an alternate (daemon-spawn instead of daemon) gem for script/delayed_job. Perhaps you can try that. And, yes, you can add the capistrano after:xxx hooks at the end of the Capistrano file.
  9. Freya Njord
    Freya Njord May 06, 2009 at 11:04 PM
    How do you get the job to actually run at the specified runs_at time? I am only working in development mode. The only way I can get them to run at all is to rake jobs:work, which doesn't pay any attention to the runs_at datetime in db. Any thoughts? Also, can you pass the runs_at param through the send_later method? That would be cool.
  10. Amit Mathur
    Amit Mathur May 08, 2009 at 2:14 AM
    @Freya: I have updated the article to show how you can specify run_at (look near the end of the article under "further tips and tricks"). rake jobs:work will also honor that. Unfortunately, there is no way to specify run_at using send_later.
  11. Justin Britten
    Justin Britten May 15, 2009 at 5:55 AM
    Thanks for a great post. All was very helpful, except I couldn't get your daemon scripts to work. I think there's a bug in your script where you manipulate ARGV which results in ARGV.first (what is used to set RAILS_ENV) being incorrectly defined. With the RAILS_ENV improperly defined, the "require File.join('config', 'environment')" statement was dumping a bunch of errors into the tmp/pids/job_runner.log file. Here's a Gist of my working daemon script: http://gist.github.com/111998 @Ryan - This may fix your problem as well. Before I made this change I was getting "LoadError: no such file to load -- daemons" in the job_runner.log file, as well as a bunch of other errors related to the fact that loading the Rails environment was failing.
  12. Shai
    Shai May 22, 2009 at 10:03 AM
    I was able to get this to work by just installing the collective_idea branch of the plug-in. My only concern thus far is that the background process is consuming a HUGE amount of resources on my slice @ Slicehost. Anyone else having this issue? Any idea to resolve it?
  13. Hi
    Hi Nov 24, 2009 at 6:39 PM

    Hi Amit, can you provide me some help ,how can i run multiple worker for delayed job with merb framework

  14. alex goji
    alex goji Feb 12, 2010 at 2:13 AM

    I installed DJ in my production machine and everything fine but after a while delayed job processing stops forever. there is no relevant log for that. PID is still remain. I don’t know why it happen and also i don’t know how to fix that. DJ do critical jobs for me if it execute jobs at their specific “run_at” time. I need solution that prevent this situation or at least restart DJ automatically when it happen

  15. milkfilk
    milkfilk May 16, 2010 at 7:34 AM

    Thanks for the write-up. FYI The Send link’s anchor has change to http://www.ruby-doc.org/core/classes/Object.html#M000332

  16. GlennB
    GlennB Oct 01, 2010 at 12:54 AM
    I substituted 'stage' for 'rails_env' and it worked.
  17. Amit Yeolekar
    Amit Yeolekar Dec 28, 2010 at 7:25 PM
    Can u please brief about running multiple jobs parallely using delayed jobs Or any alternative for the same
  18. Nathan
    Nathan Jan 04, 2011 at 1:23 AM
    can anyone tell me what if my delayed-job stop working and I have to restart manually like after 3 days then how I will recover my documents date or anything that I missed...any ideas...???
  19. http://akmathur.myopenid.com
    http://akmathur.myopenid.com Jan 06, 2011 at 4:17 PM
    @Amit Yeolekar: delayed_job just stores the jobs in a table in the database. So, you can execute them in parallel if you want.

    @Nathan: you probably mean your daemon which was executing the delayed job died and you want to run the pending tasks manually. No problem with that. Your tasks are probably still there in the DB, just go ahead and run them. You can check the table delayed_job in DB (or Delayed::Job.count from console) to see how many pending tasks you have. Also, you should use some sort of monitor to alert you if the daemon dies again in future. http://scoutapp.com is one option.
  20. Alik
    Alik Feb 01, 2011 at 4:12 AM
    I ran "ruby script/delayed_job start" but the process was crashing right away
    (you can check if process is running by running "ruby script/delayed_job status" command)
    So next I tried ruby "script/delayed_job start -t"
    The "-t" switch keeps the script running on top and does not daemonize it
    So I got to see this error message : <internal:lib/rubygems/custom_require>:29:in `require': no such file to load -- ../config/environment (LoadError)

    I fixed the problem by changing this line:
    " require File.join('config', 'environment')" to
    "require File.join('~/<project_dir>/<project_name>/config', 'environment')"
  21. wilter
    wilter Jun 27, 2011 at 7:24 AM
    If you get error: no such file to load daemon-spawn

    Rails 2.3.11:
    In environment.rb
    config.gem 'daemon-spawn', :lib=>'daemon-spawn'

    In script/delayed_job
    require 'daemon_spawn' instead 'daemon-span'

    Just my 2 cents, it worked for me.

Post a comment