Rolling Restart for Thin Cluster via Capistrano

I was recently trying to figure out the best way to do a rolling restart of a cluster of Thin instances via Capistrano so as to allow for code launches without downtime. My cluster is running behind nginx, which is just providing load balancing like so:

    # Production
    upstream thin_production_cluster {

      server unix:/tmp/thin.production_1.0.sock;
      server unix:/tmp/thin.production_1.1.sock;

      server unix:/tmp/thin.production_2.0.sock;
      server unix:/tmp/thin.production_2.1.sock;

      server unix:/tmp/thin.production_3.0.sock;
      server unix:/tmp/thin.production_3.1.sock;

    }

As you can see, I’ve broken my cluster into 3 subclusters: production_1, production_2, and production_3. This allows them to be restarted one at a time via the following Capistrano task:

  desc "Restart the application server cluster"
  task :restart_app, :roles => :app do

    p "Detecting Thin clusters..."
    num_clusters = 0
    run "#{try_sudo} ls /etc/thin | grep production.*yml | wc -l" do |ch, stream, data|
      if stream == :err
        raise "Error detecting clusters!"
      elsif stream == :out
        num_clusters = data.to_i
      end
    end

    p "Initiating rolling restart for #{num_clusters} clusters.."

    (1..num_clusters).each do |n|
      p "Restarting cluster ##{n}..."
      run "#{try_sudo} thin stop -C /etc/thin/production_#{n}.yml"
      run "#{try_sudo} thin start -C /etc/thin/production_#{n}.yml"
      sleep 5
    end

    p "Rolling restart complete."
  end

The nice thing about this approach is that so long as you stick to the naming convention for Thin config files (production_N.yml), no changes need to be made to the Capistrano task if you add/remove subclusters. Changes do unfortunately need to be made to the nginx.conf though.

Uncategorized

January 17, 2011

Leave a comment

Interesting Reading From The Last Week

The Undecidability of the Generalized Collatz Problem

The Collatz problem, widely known as the 3x + 1 problem, asks whether or not a certain simple iterative process halts on all inputs.We build on earlier work by J. H. Conway, and show that a natural generalization of the Collatz problem is recursively undecidable.

Related: Collatz Ecologies

 

Theorems For Free!

From the type of a polymorphic function we can derive a theorem that it satisfies. Every function of the same type satisfies the same theorem. This provides a free source of useful theorems, courtesy of Reynolds’ abstraction theorem for the polymorphic lambda calculus.

Related: (Even More) Theorems For Free? and Djinn, a Theorem Prover in Haskell, for Haskell

 

Object Scopes in Rails 3.1

Today I want to talk about a very tiny change I’ve made in Rails 3.1. The change was only one line. The change is subtle, but I think it has a large impact on the way we deal with scopes in Rails.

 

reading list

November 30, 2010

Leave a comment

Curried Y-Combinator for Ruby

Y-Combinators are incredibly useful functional programming tools. They allow for anonymous recursions, amongst other things. The common implementation I see in ruby is something like this:

 

  def y(&func)
    ->(x){
      ->(*args){ func[ x[x] ][*args] }
    }[ ->(x){
      ->(*args){ func[ x[x] ][*args] }
    }]
  end

 

(The above has been adapted from here)

This is then used, in the canonical factorial example, as:

 

factorial = y { |cb| ->(n) { n.zero? ? 1 : n * cb[n - 1] } }

 

This works fine, but it has that extra lambda in there, which is really just boilerplate and adds a lot of verbosity. It can be cleaned up with some currying:

 

  def ugly_y(&func)
    ->(x){
      ->(*args){ func[ x[x] ][*args] }
    }[ ->(x){
      ->(*args){ func[ x[x] ][*args] }
    }]
  end

  def y(&func)
    ugly_y { |cb| ->(*args) { func[cb, *args] } }
  end

 

Which can then be used like so:

 

factorial = y { |cb, n| n.zero? ? 1 : n * cb[n - 1] }

 

Much cleaner in my opinion.

 

functional programming, ruby

November 22, 2010

Leave a comment

Adding bounds checking to tipsy.js

Tipsy.js (homepage, github) is a great little jquery plugin for generating pretty tooltips. The problem is that it doesn’t support window bounds checking. This means that items with tooltips near the edges of the screen will often display those tooltips partially outside of the viewing pane, like so:

 

 

Thankfully, this is easy enough to fix. Simply add the following function to the end of jquery.tipsy.js :

 

$.fn.tipsy.autoBounds = function(margin, prefer) {
	return function()
	{
		var dir = { ns: prefer[0], ew: (prefer.length > 1 ? prefer[1] : false) }

		bound_top = $(document).scrollTop() + margin;
		bound_left = $(document).scrollLeft() + margin;

		if ( $(this).offset().top < bound_top) {
			dir.ns = 'n';
		}
		if ( $(this).offset().left < bound_left ) {
			dir.ew = 'w';
		}
		if ( $(window).width() + $(document).scrollLeft() - $(this).offset().left < margin ) {
			dir.ew = 'e';
		}
		if ( $(window).height() + $(document).scrollTop() - $(this).offset().top < margin ) {
			dir.ns = 's';
		}

		return dir.ns + (dir.ew ? dir.ew : '');
	}
};

 

This adds a new parameterizable “auto gravity” function, like the built in autoNS and autoEW functions. It takes a margin that it uses to choose the safe distance from the edge of the screen, and if any elements are within that margin it adjusts the direction (“gravity”, in tipsy parlance) of their tooltips to ensure that they don’t go out of bounds. It also takes a preferred gravity that it will try to deviate from minimally in laying out tooltip directions.

Once this is in place, it’s supplied as a gravity function more or less like the standard ones:

 

$('a[rel=tipsy]').tipsy({ gravity: $.fn.tipsy.autoBounds(150, 'n') });

 

And with that, tooltips are laid out in a manner that keeps them in the viewable region:

 

 

A fork of tipsy with this modification already in place can be found at my github, here.

javascript

November 21, 2010

Leave a comment