af83

Release: Sponges, daemons in a pool

When I build workers, I want them to be like an army of little spongebobs, always on the edge and ready to work. sponges helps you create this army of sponges, control them, and well…kill them at will too. Making them eager to work is now your job. :)

Basically, sponges is a ruby supervisor that forks processes and controls their execution, termination and forks a new process each time a process disappears from the processes pool.

For example, the following command will start a supervision daemon and 8 processes of "a_worker".

ruby a_worker.rb start -d -s 8

If you kill the supervisor, it will cleanly terminate the child processes.

Installation

Ruby 1.9.3 (or superior).

Install it with rubygems:

gem install sponges

With bundler, add it to your Gemfile:

gem "sponges"

Usage

In a file called example.rb:

# The worker class is the one you want to daemonize.
#
require 'sponges'

class Worker
  def initialize
    # Trap the HUP signal, set a boolean to true.
    trap(:HUP) {
      Sponges.logger.info "HUP signal trapped, clean stop."
      @hup = true
    }
  end

  def run
    Sponges.logger.info Process.pid
    if @hup # is true, we need to shutdown this worker
      Sponges.logger.info "HUP signal trapped, shutdown..."
      exit 0 # everything's fine, we can exit
    else # this worker can continue its work
      sleep rand(20)
      run
    end
  end
end

Sponges.configure do |config|
  config.logger            = MyCustomLogger.new   # optionnal
  config.size              = 3                    # optionnal, default to cpu's size
  config.daemonize         = true                 # optionnal, default to false
  config.port              = 5032                 # optionnal, default to 5032
  config.after_fork do
    puts "Execute code when a child process is created"
  end
  config.on_chld do
    puts "Execute code when a child process is killed"
end

# Register a pool named "worker_name".
#
Sponges.start "worker_name" do
  Worker.new({some: args}).run
end

See the help message :

ruby example.rb

Start workers :

ruby example.rb start

Start workers and daemonize them:

ruby example.rb start -d

Start 8 instances of the worker and daemonize them:

ruby example.rb start -d -s 8 # By default, size equals cpu core's size.

Restart gracefully 4 instances of the worker, with a timeout of 3 seconds and daemonize them:

ruby example.rb restart -g -s 4 -t 3

Stop workers with a QUIT signal :

ruby example.rb stop

Stop workers with a KILL signal :

ruby example.rb kill

Stop workers with a HUP signal :

ruby example.rb stop -g -t 5

In this case, you will have to trap the HUP signal, and handle a clean stop from each worker. The point is to wait for a task to be done before quitting. A timeout can be specified with the -t option. When this timeout is hit, the process will be automatically killed.

Increment worker's pool size :

ruby example.rb increment # will add a worker to the pool.

Decrement worker's pool size :

ruby example.rb decrement # will remove a worker to the pool.

Http supervision

sponges provides an http interface to supervise pool's activity, and to expose pids. Http supervision can be enabled in configuration:

Sponges.configure do |config|
  config.port            = 3333
end

By default, sponges listens on port 5032, and responds in json. Here is an example of response:

{
  "supervisor":{
    "pid":11537,
    "pctcpu":0.0,
    "pctmem":0.22,
    "created_at":"2013-03-05 15:21:04 +0100"
  },
  "children":[
    {
      "pid":11540,
      "pctcpu":0.0,
      "pctmem":0.21,
      "created_at":"2013-03-05 15:21:04 +0100"
    },
    {
      "pid":11543,
      "pctcpu":0.0,
      "pctmem":0.21,
      "created_at":"2013-03-05 15:21:04 +0100"
    },
    {
      "pid":11546,
      "pctcpu":0.0,
      "pctmem":0.21,
      "created_at":"2013-03-05 15:21:04 +0100"
    },
    {
      "pid":11549,
      "pctcpu":0.0,
      "pctmem":0.21,
      "created_at":"2013-03-05 15:21:04 +0100"
    }
  ]
}

Patterns

We use sponges on several of our projets, and we can see two different patterns of use for sponges.

The classic way

A worker build upon Redis, Beanstalkd, or anything capable of blocking on a queue. This worker waits for a job, receive a job, process the job, and waits for a new job. Simple and straightforward.

This task must use pull/push socket, and not pub/sub. When a pool of workers is starting, we do not want to have several workers receiving the same task.

The brutal way

The brutal way looks like the classic way, except it does not wait for a new job, it exits.

Yes, exactly, it exits. By doing so, the supervisor will catch a signal for an exiting child, and fork a new process. This process takes the turn on the queue, and wait for a job.

Why this pattern ? Because of memory management. By exiting, the memory is completely freed by the kernel. We used this pattern in a worker manipulating some huge xml files (around 1 go) which happens to leak memory quite fast.

However, this pattern is slower, since it has to require everything on each job.

Unix

Internally, sponges strongly relies on Unix, using fork, signals and pipe.

fork is used to create one or several worker process.

Signals are used to delegate commands from the supervisor to each process. By nature, signals are asynchronous and unidirectional. Not knowing when a signal occurs is not a problem, but not knowing when Ruby will run a signal handler could be one. The tricky case: receiving a signal while doing some work on another signal handler.

We solved these problems with a pipe. When a signal occurs, only one action is performed: writing this signal to a pipe. A thread dedicated to signals handlers reads this pipe and performs handler one at the time. This solves two problems:

  • by enforcing signals to be synchronous, we erase most of weird edge cases we have seen in the firsts versions of Sponges.
  • Ruby 2.0 compatibility. From 2.0, Ruby does not allow to use a mutex in a signal handler. So something like writing to a logger could break. Writing to a pipe, on the other hand, does not break.

One last word

Sponges would not have been the same without Jesse Storimer and his awesome book about Unix.

blog comments powered by Disqus