rrichards
2/4/2015 - 8:43 PM

README.md

README.md
content_copyfile_download
  • Rendered
  • Source

Tutum Zero Downtime Re-deploy

I tweeted Tutum last night asking if they're looking at implementing zero downtime re-deploys for a given service. Slightly surprised by their response as it seems like a critical feature if you want to use the service for a production environment.

"not a top priority, but by Spring :)"

As Tutum currently doesn't support graceful termination of containers within a service, I was experiencing a 5-10 second window of 503 errors, so decided to use the following hack (code below) until the feature is officially implemented.

Solution

  1. Create two identical web app services.
  2. Link HAProxy service to both.
  3. Redeploy one service successfully.
  4. Wait 10 seconds.
  5. Redeploy the second service.

Note - use Tutum's HAProxy image so that it re-configures to new containers, and set the POLLING_PERIOD environment variable to 1.

Example Tutum Setup

Notes

  • Could instead have a single service and redeploy the individual containers.
  • The code could be better, would be better in a Thor app.
  • Tutum is awesome!
tutum_zero_downtime.rb
content_copyfile_download
require 'tutum'

module TutumHack
  class ZeroDowntime
    def initialize(service_names)
      @service_names = service_names
    end

    def start!
      ids.each do |id|
        redeploy id
        until service_running? id
          print '.'
          sleep 1
        end
        puts 'Finished!'
        sleep 10
      end 
    end

    private

    attr_reader :service_names

    API_KEY  = '123456789'
    USERNAME = 'batman'

    def ids
      services.map do |service|
        service['uuid'] if known_service? service['name']
      end.compact
    end

    def known_service?(name)
      service_names.include? name
    end

    def redeploy(id)
      print "Re-deploying: #{id}"
      tutum.services.redeploy id
    end

    def service(id)
      tutum.services.get id
    end

    def services
      tutum.services.list['objects']
    end

    def service_running?(id)
      service(id)['state'] == 'Running'
    end

    def tutum
      @tutum ||= Tutum.new(USERNAME, API_KEY) 
    end
  end
end

service_names = ['batcomputer-1', 'batcomputer-2']

TutumHack::ZeroDowntime.new(service_names).start!