Start and Stop tasks for resque workers, with capistrano deploy hook, killing correct processes (without God)

Console Output

 ** [out :: ec2-174-129-60-95.compute-1.amazonaws.com] $ kill -s QUIT 12689 12692 12696 12686
 ** [out :: ec2-174-129-60-95.compute-1.amazonaws.com] 
 ** [out :: ec2-174-129-60-95.compute-1.amazonaws.com] Starting 1 worker(s) with QUEUE: *
 ** [out :: ec2-174-129-60-95.compute-1.amazonaws.com] 
 ** [out :: ec2-174-129-60-95.compute-1.amazonaws.com] Starting 3 worker(s) with QUEUE: OMG
 ** [out :: ec2-174-129-60-95.compute-1.amazonaws.com] 

deploy.rb

after "deploy:symlink", "deploy:restart_workers"

##
# Rake helper task.
# http://pastie.org/255489
# http://geminstallthat.wordpress.com/2008/01/27/rake-tasks-through-capistrano/
# http://ananelson.com/said/on/2007/12/30/remote-rake-tasks-with-capistrano/
def run_remote_rake(rake_cmd)
  rake_args = ENV['RAKE_ARGS'].to_s.split(',')

  cmd = "cd #{fetch(:latest_release)} && bundle exec #{fetch(:rake, "rake")} RAILS_ENV=#{fetch(:rails_env, "production")} #{rake_cmd}"
  cmd += "['#{rake_args.join("','")}']" unless rake_args.empty?
  run cmd
  set :rakefile, nil if exists?(:rakefile)
end

namespace :deploy do
  desc "Restart Resque Workers"
  task :restart_workers, :roles => :worker do
    run_remote_rake "resque:restart_workers"
  end
end

resque.rake

require 'resque/tasks'

namespace :resque do  
  task :setup => :environment

  desc "Restart running workers"
  task :restart_workers => :environment do
    Rake::Task['resque:stop_workers'].invoke
    Rake::Task['resque:start_workers'].invoke
  end
  
  desc "Quit running workers"
  task :stop_workers => :environment do
    stop_workers
  end
  
  desc "Start workers"
  task :start_workers => :environment do    
    run_worker("*", 1)
    run_worker("OMG", 3)
  end
  
  
  
  def store_pids(pids, mode)
    pids_to_store = pids
    pids_to_store += read_pids if mode == :append

    # Make sure the pid file is writable.    
    File.open(File.expand_path('tmp/pids/resque.pid', Rails.root), 'w') do |f|
      f <<  pids_to_store.join(',')
    end
  end

  def read_pids
    pid_file_path = File.expand_path('tmp/pids/resque.pid', Rails.root)
    return []  if ! File.exists?(pid_file_path)
    
    File.open(pid_file_path, 'r') do |f| 
      f.read 
    end.split(',').collect {|p| p.to_i }
  end

  def stop_workers
    pids = read_pids
    
    if pids.empty?
      puts "No workers to kill"
    else
      syscmd = "kill -s QUIT #{pids.join(' ')}"
      puts "$ #{syscmd}"
      `#{syscmd}`
      store_pids([], :write)
    end
  end
  
  # Start a worker with proper env vars and output redirection
  def run_worker(queue, count = 1)
    puts "Starting #{count} worker(s) with QUEUE: #{queue}"

    ##  make sure log/resque_err, log/resque_stdout are writable.
    ops = {:pgroup => true, :err => [(Rails.root + "log/resque_err").to_s, "a"], 
                            :out => [(Rails.root + "log/resque_stdout").to_s, "a"]}
    env_vars = {"QUEUE" => queue.to_s, 'RAILS_ENV' => Rails.env.to_s}

    pids = []
    count.times do
      ## Using Kernel.spawn and Process.detach because regular system() call would
      ## cause the processes to quit when capistrano finishes
      pid = spawn(env_vars, "rake resque:work", ops)
      Process.detach(pid)
      pids << pid
    end
    
    store_pids(pids, :append)
  end
end

I used this as the base of our task, but we found that spawning processes to run Rake (and Rails) took too long, and was far too CPU intensive, so I updated it to fork the workers directly from the already running Rake task.

https://gist.github.com/2371233

I get the following error "`deploy:restart_workers' is only run for servers matching {:roles=>:worker}, but no servers matched". Any hits?

That sounds like a capistrano error.

At the top of your Capfile you define servers and their role, try

role :worker, "worker1.myhostname.com"

Yap, I needed to declare that role beforehand, thanks, great gist

…

On May 14, 2012 2:06 AM, "Mal Curtis" < ***@***.***> wrote: That sounds like a capistrano error? --- Reply to this email directly or view it on GitHub: https://gist.github.com/1870642