Logo

Testing IP Whitelisting in your Specs and Features

3 months ago | Alex Rothenberg: Common Sense Software

Rails has so much support for testing built into itself that its rare I come up with something that’s hard to test but HTTP headers is not easy. Normally you don’t have to worry about HTTP headers as they’re set by the browser and you don’t do much with them. Recently I was working on an application where each user has an IP whitelist and they are only allowed to come from their whitelisted IP addresses. This isn’t as crazy as it sounds since the app is in a corporate environment and the users will all be coming from their corporate networks.

Basically this means our authentication method needs 3 pieces of information

  1. username
  2. password
  3. remote ip address

What makes this interesting is that the first two are input by the user but the ip address comes from the browser and network. Writing an RSpec unit test or Cucumber scenario to test user parameters (username and password) is something we’ve all done before but today I’m going to talk about how you can also test the IP address in a header.

Implementation

Before we look at how to test this let’s take a look at the implementation of our SessionController.

class SessionsController < ApplicationController
  def new
    @session = Session.new
  end

  def create
    remote_ip_address = request.headers['X-Forwarded-For'] || request.headers['REMOTE_ADDR']
    @session = Session.create(params[:username], params[:password], remote_ip_address)

    if @session.valid?
      session[:current_user] = @session.user
      redirect_to root_url
    else
      flash.now[:error] = 'Unable to authenticate. Please try again'
      render :new
    end
  end

  def destroy
    session[:current_user] = nil
    redirect_to session
  end
end

These three actions provide login and logout.

  • new displays the login form with username & password fields
  • create uses the username and password from the form as well as the ip address to create a session (i.e. authenticate). In case the request hops through some proxy servers we use the X-Forwarded-For header to get the source IP and not the proxy’s IP.
  • destroy users need to log out (but we wont talk about that anymore here)

This works, but you shouldn’t trust me. We need tests around the create action!

Unit Testing the IP Whitelist with RSpec

Our Controller Spec needs to pass all 3 pieces of information (username, password & ip address) to the controller. Passing the username and password is pretty standard and something I’m sure you’ve done before. They come from a form so we pass them as a hash in the second argument to post.

post :create, {:username => 'alex', :password => 'secret'}

Unfortunately we can’t pass the IP the same way because the post method in ActionController::TestCase doesn’t support passing headers in (but it does take the session or flash - that’s interesting to remember for some other time).

def post(action, parameters = nil, session = nil, flash = nil)
  process(action, parameters, session, flash, "POST")
end

If we keep looking around it turns out the ActionDispatch::TestRequest object has a nice convenience method that lets us specify the remote_addr directly.

def remote_addr=(addr)
  @env['REMOTE_ADDR'] = addr
end

If we add a line to our spec we can handle the case where the IP comes in the REMOTE_ADDR HTTP header.

request.remote_addr = '192.168.1.100'
post :create, {:username => 'alex', :password => 'secret'}

We still need to deal with the X-Forwarded-For case. While Rails doesn’t give us a convenience method, by looking at the implementation of the remote_addr= method we can see how to set this header ourselves.

request.env['X-Forwarded-For'] = '192.168.1.100'
post :create, {:username => 'alex', :password => 'secret'}

Putting it all together we end up with a controller spec that looks like this.

require 'spec_helper'

describe SessionsController do
  describe '#create' do
    describe 'successfully' do
      let(:alex) { mock }
      let(:valid_session) { mock(:valid? => true, :user => alex )}
      before do
        Session.should_receive(:create).with('alex', 'secret', '192.168.1.100').and_return(valid_session)
      end
      describe 'using REMOTE_ADDR' do
        before do
          request.remote_addr = '192.168.1.100'
          post :create, {:username => 'alex', :password => 'secret'}
        end
        it { should redirect_to root_path }
        it { should set_session(:current_user).to(alex)}
      end
      describe 'using X-Forwarded-For' do
        before do
          request.remote_addr = '172.16.254.1'
          request.env['X-Forwarded-For'] = '192.168.1.100'
          post :create, {:username => 'alex', :password => 'secret'}
        end
        it { should redirect_to root_path }
        it { should set_session(:current_user).to(alex)}
      end
    end

    describe 'unsuccessfully' do
      let(:invalid_session) { mock(:valid? => false) }
      before do
        Session.should_receive(:create).with('alex', 'secret', '192.168.1.100').and_return(invalid_session)
      end
      describe 'using REMOTE_ADDR' do
        before do
          request.remote_addr = '192.168.1.100'
          post :create, {:username => 'alex', :password => 'secret'}
        end
        it { should render_template :new }
      end
      describe 'using X-Forwarded-For' do
        before do
          request.remote_addr = '172.16.254.1'
          request.env['X-Forwarded-For'] = '192.168.1.100'
          post :create, {:username => 'alex', :password => 'secret'}
        end
        it { should render_template :new }
      end
    end
  end
end

To sum up we can

  • pass parameters as a hash in the post method

    post :create, {:username => 'alex', :password => 'secret'}

  • set the remote_addr on the request with a convenience method

    request.remote_addr = '192.168.1.100'

  • et the X-Forwarded-For directly on the requests’s environment hash

    request.env['X-Forwarded-For'] = '192.168.1.100'

Integration Testing the IP Whitelist in a Cucumber Feature

We face a similar issue when writing our cucumber scenarios - its easy to pass the username and password but harder to pass the IP address. The solution turns out to be similar but not quite exactly the because our Cucumber steps will use Capybara instead of ActionController::TestCase directly. Before we look into how to implement the steps, let’s write the feature we want which will help us define the steps we need.

Feature: Authentication of a user
  In order to ensure a really secure application
  As a user
  I want my IP address to be validated during login

  Background:
    Given the following user exists:
      | username | password | company                   |
      | alex     | secret   | ip_address: 192.168.1.100 |

  Scenario: Successful log in
    Given I am connecting from ip "192.168.1.100"
     When I log in as "alex" with password "secret"
     Then I should be on the home page

  Scenario: Successful log in with X-Forwarded-For header
    Given I am connecting from ip "192.168.1.100" behind a proxy
     When I log in as "alex" with password "secret"
     Then I should be on the home page

  Scenario: Failed log in from wrong IP
    Given I am connecting from ip "172.16.254.1"
     When I log in as "alex" with password "secret"
     Then authentication should have failed

  Scenario: Failed log in from wrong IP behind a proxy
    Given I am connecting from ip "172.16.254.1" behind a proxy
     When I log in as "alex" with password "secret"
     Then authentication should have failed

We immediately realize we don’t know how to write the first step

Given /^I am connecting from ip "([^"]*)"$/ do |ip_address|
  pending # How do we set the IP Address???
end

To figure this out we need to dig into how capybara works.

We don’t call post in ActionController::TestCase directly instead letting capybara do it for us. To see what capybara is doing we can skip that step and implement the login step

Given /^I am connecting from ip "([^"]*)"$/ do |ip_address|
  # do nothing for now
end

When /^I log in as "([^"]*)" with password "([^"]*)"$/ do |name, password|
  visit(new_session_path)
  fill_in('User name', :with => name)
  fill_in('Password', :with => password)
  click_button('Log In')
end

and edit the SessionsController to show us the stack trace.

class SessionsController < ApplicationController
  def create
    raise caller.inspect
  end
end

The stack trace is very big but if we look closely, somewhere in the middle of it we see lines below that show how capybara uses the rack-test gem to submit our form.

~/.rvm/gems/ruby-1.8.7-p334/gems/rack-test-0.6.1/lib/rack/test.rb:66:in `post'
~/.rvm/gems/ruby-1.8.7-p334/gems/capybara-1.1.2/lib/capybara/rack_test/browser.rb:62:in `send'
~/.rvm/gems/ruby-1.8.7-p334/gems/capybara-1.1.2/lib/capybara/rack_test/browser.rb:62:in `process'
~/.rvm/gems/ruby-1.8.7-p334/gems/capybara-1.1.2/lib/capybara/rack_test/browser.rb:27:in `submit'
~/.rvm/gems/ruby-1.8.7-p334/gems/capybara-1.1.2/lib/capybara/rack_test/form.rb:64:in `submit'
... more lines omitted...
~/.rvm/gems/ruby-1.8.7-p334/gems/capybara-1.1.2/lib/capybara/node/actions.rb:38:in `click_button'

Looking at the Rack::Test#post method we see something similar to what we saw before in ActionController::TestCase but its not quite identical. It takes the env as a parameter so we need to figure out how to inject our header in there.

def post(uri, params = {}, env = {}, &block)
  env = env_for(uri, env.merge(:method => "POST", :params => params))
  process_request(uri, env, &block)
end

Following the stack trace up we see the env passed into Rack::Test::Session.post comes from Capybara::RackTest::Browser and it turns out that env is computed in the Capybara::RackTest::Browser#env method.

def options
  driver.options
end

def env
  env = {}
  begin
    env["HTTP_REFERER"] = last_request.url
  rescue Rack::Test::Error
    # no request yet
  end
  env.merge!(options[:headers]) if options[:headers]
  env
end

The key is in the line env.merge!(options[:headers]) if options[:headers] and those options are delegated to the driver. Now we know how to inject our IP address onto the driver’s options.

Given /^I am connecting from ip "([^"]*)"$/ do |ip_address|
  page.driver.options[:headers] = {'REMOTE_ADDR' => ip_address}
end

Putting it all together we can write all our steps

Given /^I am connecting from ip "([^"]*)"$/ do |ip_address|
  page.driver.options[:headers] = {'REMOTE_ADDR' => ip_address}
end

Given /^I am connecting from ip "([^"]*)" behind a proxy$/ do |ip_address|
  page.driver.options[:headers] = {'X-Forwarded-For' => ip_address}
end

When /^I log in as "([^"]*)" with password "([^"]*)"$/ do |name, password|
  visit(new_session_path)
  fill_in('User name', :with => name)
  fill_in('Password', :with => password)
  click_button('Log In')
end

Then /^I should be on the home page$/ do
  URI.parse(current_url).path.should == root_path
end

Then /^authentication should have failed$/ do
  page.text.should include 'Unable to authenticate. Please try again'
end

Now the scenarios we wrote before all pass.

To sum up

  • capybara handles form submission superbly with

    fill_in('User name', :with => name)

    click_button('Log In')

  • we can set any HTTP header in capybara with

    page.driver.options[:headers] = {'REMOTE_ADDR' => ip_address}

Testing is good

Since we’re testing the IP logic at both the unit level with RSpec and integration level with Cucumber and Capybara we can be pretty sure it’s all going to work correctly.

Programming With Kids

3 months ago | Alex Rothenberg: Common Sense Software

I’ve started to teach my kids to program. I figured I build websites professionally and it’d be a fun way for me to share what I do and help supplement their learning. And it was something they expressed interest in not something I was pushing. I suggested we build our own small version of facebook or twitter. Very quickly I learned two truths

  1. websites are boring
  2. games are fun

Okay. I’ve never built a game before after a little digging there are plenty of tools in the open source world and many built on Ruby. We’re currently experimenting with three different tools/technologies.

Shoes

I first came across Shoes several years ago and was blown away. It was originally written by why and is now maintained by Team Shoes on github. “Shoes is the best little DSL for cross-platform GUI programming there is. It feels like real Ruby, rather than just another C++ library wrapper”

Writing a Shoes app feels just like writing a Ruby app (it is Ruby!). The best analogy I can use is that what Rails does for websites, Shoes does for GUI apps.

If you want to create a blue rectangle on a page, here’s your app

Shoes.app do
  fill blue
  rect :top => 25, :left => 50, :height => 75, :width => 150
end

A blue rectangle

We can make it a bit more interactive and allow the user to move our rectangle around with the arrow keys and display the current coordinates

Shoes.app do
  fill blue
  @player = rect :top => 25, :left => 50, :height => 75, :width => 150
  @current_coordinates = para "(#{@player.left}, #{@player.top})"

  keypress do |key|
    @player.left += 10 if key == :right
    @player.left -= 10 if key == :left
    @player.top  += 10 if key == :down
    @player.top  -= 10 if key == :up
    @current_coordinates.replace "(#{@player.left}, #{@player.top})"
  end
end

The Blue Rectangle Moves

We’re not limited to blue rectangles. We can replace it with an image

Shoes.app do
  @player = image 'images/Starfighter.png', :top => 25, :left => 50
  @current_coordinates = para "(#{@player.left}, #{@player.top})"

  keypress do |key|
    @player.left += 10 if key == :right
    @player.left -= 10 if key == :left
    @player.top  += 10 if key == :down
    @player.top  -= 10 if key == :up
    @current_coordinates.replace "(#{@player.left}, #{@player.top})"
  end
end

The Player is a Starship

We’re writing Ruby in a fairly natural way. Shoes just gives us GUI methods like rect to create a rectangle or para to create a paragraph. Because this is Ruby, as your Shoes app gets more complex you can create classes and methods to organize and keep it manageable just as you would in any other app.

There are all sorts of great resources out there including

Gosu

Gosu is a gaming library so while Shoes lets us build any sort of GUI apps this is seemed like it might be a better fit since we’re interested in gaming. Luckily there’s a gem that wraps up the Ruby interface (Gosu can be used from C++ or Ruby).

gem install gosu

If we want to build a similar game to what we did in Shoes with a play we move around via the arrow keys we need to subclass the Gosu::Window class.

require 'rubygems'
require 'gosu'

class Player
  def initialize(window)
    @image = Gosu::Image.new(window, "media/Starfighter.png", false)
    @x, @y = 125, 50
    @angle = 0.0
  end

  def draw
    @image.draw_rot(@x, @y, 0, @angle)
  end
end

class GameWindow < Gosu::Window
  def initialize
    super(640, 480, false)
    self.caption = "Gosu Tutorial Game"
    @player = Player.new(self)
  end

  def draw
    @player.draw
  end
end

window = GameWindow.new
window.show

And we see a window with a player that looks like a starship

Starship player

Its not too hard to make it move by overriding the update method in our window class

require 'rubygems'
require 'gosu'

class Player
  attr_accessor :x, :y

  def initialize(window)
    @image = Gosu::Image.new(window, "media/Starfighter.bmp", false)
    @x, @y = 75, 50
    @angle = 0.0
  end

  def draw
    @image.draw_rot(@x, @y, 0, @angle)
  end
end

class GameWindow < Gosu::Window
  def initialize
    super(400, 300, false)
    self.caption = "Our Game"
    @player = Player.new(self)
    @current_coordinates = Gosu::Font.new(self, Gosu::default_font_name, 20)
  end

  def update
    @player.x -= 10 if button_down? Gosu::KbLeft
    @player.x += 10 if button_down? Gosu::KbRight
    @player.y += 10 if button_down? Gosu::KbDown
    @player.y -= 10 if button_down? Gosu::KbUp
  end

  def draw
    @player.draw
    @current_coordinates.draw("(#{@player.x}, #{@player.y})", 10, 10, 0, 1.0, 1.0, 0xffffff00)
  end
end

window = GameWindow.new
window.show

Starship player moves

This example can be extended into a full Asteroids like like game where your ship has inertia. You should look at the source or an explanation on the gosu site.

There are all sorts of great resources out there including

Even though we’re still writing Ruby, Gosu feels more like C++ Windows development I used to do long long time ago. I’m not sure if that’s inevitable and need to keep using Gosu to find out.

gamesalad

The last framework we’ve been working with is pretty different. GameSalad advertises it lets you “Create games for iPhone, iPad, Android, Mac, and HTML5. No coding required.”

It follows a model similar to what Adobe Flash uses where you have Scenes containing Actors. You write your programs in a visual editor by dragging and dropping Actors onto Scenes, Rules onto Actors and Behavior onto Rules. For instance if we have a starship actor and we drop these rules onto it

Starfighter rules

We will get our familiar spaceship that can move left and right

Starfighter Moving

GameSalad is the least familiar to me but seems to be easiest for my kids to start working on. Not having to write any “code” or “do programming” makes it much easier to get started. It also can create iPhone or iPad games and I would never dream of exposing my kids to Objective-C.

What’s next?

We’ve started experimenting with all three of these tools and so far are having fun with all three. Hopefully we’ll figure out what works for us and perhaps try to write about it again in a few months.

After writing this I came across a recent NY Times article Programming for Children, Minus Cryptic Syntax and Scratch also sounds interesting so I may have to look into that sometime too.

Twitter is the new RSS Reader

4 months ago | Alex Rothenberg: Common Sense Software

In 2008 I thought RSS was an awesome way to stay abreast of what’s going on, but now its 2011 and I find myself using Twitter more often than Google Reader to find new and interesting articles people have written. Readers tweet and retweet articles they find interesting which seems a lower barrier than leaving an “I like this” comment. As an author Twitter also gives you some idea of who is reading your posts and a way to connect with them.

Back in 2008 I created a blog aggregator site http://waywework.it to group the all the people I work with and promote others to share their thoughts. I was so excited I even wrote an article about it.

Now that its 2011, I’ve been asking myself how could I update http://waywework.it for the twitter world of today?

I decide that if we’re going to follow people on twitter that’s what my site should facilitate. When new posts come in it should tweet them letting you see them if you follow @WayWeWorkIT.

Enabling API access to twitter

Once I had this I added the twitter gem to my application. I have to give a shout out John Nunemaker for writing this fantastic gem which made my task so simple.

In the Gemfile

gem 'twitter'

I created a new twitter account @WayWeWorkIT and registered an application at https://dev.twitter.com/apps so I had my OAuth and access tokens.

The only trick was I had to go Application Settings tab and configure it for Read and Write access then regenerate the tokens.

Now that I had the keys and tokens from twitter I had to tell my application to use them without hardcoding them in my code. This took two steps. First, configuring the app to read the tokens from the environment in config/initializers/twitter.rb. Yes I am somewhat paranoid about accidentally tweeting from development but that if Rails.env.production? should save me.

if Rails.env.production?
  Twitter.configure do |config|
    config.consumer_key       = ENV['TWITTER_CONSUMER_KEY']
    config.consumer_secret    = ENV['TWITTER_CONSUMER_SECRET']
    config.oauth_token        = ENV['TWITTER_OAUTH_TOKEN']
    config.oauth_token_secret = ENV['TWITTER_OAUTH_TOKEN_SECRET']
  end
end

Secondly, setting the tokens on the heroku environment (I typed the real tokens instead of the XXXXXXXX’s).

$ heroku config:add TWITTER_CONSUMER_KEY=XXXXXXXX
$ heroku config:add TWITTER_CONSUMER_SECRET=XXXXXXXX
$ heroku config:add TWITTER_OAUTH_TOKEN=XXXXXXXX
$ heroku config:add TWITTER_OAUTH_TOKEN_SECRET=XXXXXXXX

We can test it out (after deploying with git push heroku)

$ heroku console
>> Twitter.tweet('http://waywework.it aggregates blog articles')
=> # some big object returned
>> Twitter.user_timeline('wayweworkit').first.text
=> "http://t.co/FCmUQdc3 aggregates blog articles"

Great we just tweeted our first tweet for the world to see.

Updating WayWeWork.IT to tweet new posts

The app periodically scans the rss feeds it tracks and when it sees a new post it creates it in the app’s database.

First we add a twitter_username to each feed we’re tracking

class AddTwitterUsernameToFeeds < ActiveRecord::Migration
  def change
    add_column :feeds, :twitter_username, :string
  end
end

Then, add an after_create callback to tweet each time we create a new post.

class Post < ActiveRecord::Base
  after_create :tweet

  delegate :twitter_username, :to => :feed

  def twitter_username_with_at_sign
    "@#{feed_twitter_username || 'WayWeWorkIT'}
  end

  # See https://dev.twitter.com/docs/tco-link-wrapper/faq#Will_t.co-wrapped_links_always_be_the_same_length
  # We should query instead of hardcoding 20
  def short_url_length
    20
  end

  def tweet
    if Rails.env.production?
      non_title_part_of_tweet = " #{'x'*short_url_length} via #{twitter_username_with_at_sign}"
      max_title_length = 140 - non_title_part_of_tweet.length

      tweet = "#{title.truncate(max_title_length)} #{url} via #{twitter_username_with_at_sign}"
      Twitter.update(tweet)
    end
  end
end

Again with the “if Rails.env.production?” paranoia? You do know that you can never be too paranoid :)

With the twitter gem its one line to tweet Twitter.update(tweet). The rest of it is to shorten the title so twitter’s 140 character limit wont cut off the url or the author’s name.

Once this is in we’ll start seeing tweets like

Using BDD and the email_spec gem to implement Email www.alexrothenberg.com/2011/10/31/usi… via @alexrothenberg

Twitter is the new RSS Reader http://www.alexrothenberg.com/2011/11/07/twitter-is-the-new-rss-reader.html via @alexrothenberg

Go ahead and follow @WayWeWorkIT on twitter and you’ll start seeing these blog posts.

Using BDD and the email_spec gem to implement Email

4 months ago | Alex Rothenberg: Common Sense Software

When implementing email functionality, the email_spec gem is something I’ve decided I can’t live without. It makes it so easy to write RSpec specs and Cucumber features around your email that you have no excuse not to. Today I’m going to go through an example how I recently used BDD to send an email in an app I was working on.

When I think about email and my Rails environments this is how I typically want them to behave.

  • test should not send emails and allow us to write specs or features against them
  • development should not send emails but provide a UI to view what would be sent on localhost
  • staging should not send emails but provide a UI to view what would be sent on a server
  • production should send real emails to real people

Last week I wrote about using letter opener to View Sent Email on a Server (without actually sending anything) describing how the letter_opener gem helps us in the development and staging environments. This article focuses on using email_spec gem in the test environment.

Our Sample Project

Let’s imagine we are working on a new startup in stealth mode. We want to generate buzz and prepare for a beta launch. We’re hiding the fact its all vaporware with a splashy homepage where people can request an invitation to the beta and records their email in our database. This is the same example as my article last week and there’s a live demo at http://awesome-site-staging.heroku.com/.

What we’ll do today is not just record the email address but also send a “thanks for your interest” email to the user. We’re going to do this in BDD fashion bouncing back and forth between Cucumber Scenarios and RSpec Unit Tests

  1. Writing a cucumber scenario

  2. While the scenario fails

    1. Write a failing rspec spec
    2. Write code to make it pass

The Cucumber scenario tells us what should be accomplished and when it fails it we use that to tell us what unit test we should write.

Adding the email_spec gem

We add it to our Gemfile

group :test do
  gem 'email_spec'
end

We bundle and use the email_spec generator to let it initialize itself.

$ bundle
$ rails g email_spec:steps

There’s also a manual step to get cucumber to load the email_spec gem. We need to create a file features/support/email_spec.rb

require 'email_spec/cucumber'

Our first Cucumber Scenario

We know that when a user requests an invitation they should get an email so we write that requirement as a Cucumber Scenario. features/request_an_invitation.feature

Feature: Build excitement for this vaporware
  In order to drum up interest
  As a user
  I will receive an exciting email when I request an invitation

  Scenario: Someone requests an invitation and receives an email
    Given I am on the home page
     When I request an invitation for "gullible@lemmings.com"
     Then "gullible@lemmings.com" should receive 1 email
      And they open the email
      And they should see the email delivered from "alex@awesome-startup.com"
      And they should see "Invitation request for Awesome New Startup received" in the email subject
      And they should see "Dear gullible@lemmings.com," in the email text part body
      And they should see "We have received your request " in the email text part body
      And they should see "Please check back at http://awesome-site-staging.heroku.com" in the email text part body

We run it and it tells us we have a couple of missing steps.

1 scenario (1 undefined)
9 steps (7 skipped, 2 undefined)
0m0.006s

You can implement step definitions for undefined steps with these snippets:

Given /^I am on the home page$/ do
  pending # express the regexp above with the code you wish you had
end

When /^I request an invitation for "([^"]*)"$/ do |arg1|
  pending # express the regexp above with the code you wish you had
end

Oh right, the training wheels came off in cucumber-rails v1.1.1 and we don’t have web_steps.rb anymore. Let’s write these steps using capybara in features/step_definitions/invite_steps.rb

Given /^I am on the home page$/ do
  visit root_path
end

When /^I request an invitation for "([^"]*)"$/ do |email|
  visit root_path
  fill_in 'email', :with => email
  click_button 'Request Invitation'
end

We run one more time and get a failure we expect. Its telling us we haven’t written any code to implement the scenario yet!

  Then "gullible@lemmings.com" should receive 1 email
  expected: 1
       got: 0 (using ==) (RSpec::Expectations::ExpectationNotMetError)
  ./features/step_definitions/email_steps.rb:52:in `/^(?:I|they|"([^"]*?)") should receive (an|no|\d+) emails?$/'
  features/request_an_invitation.feature:9:in `Then "gullible@lemmings.com" should receive 1 email'

Dropping into RSpec unit tests

Our failing feature tells us what we need to implement so we drop down to the unit test level and start implementing it with TDD. The feature tells us an email should be be generated so let’s go. We’ll add to our invites_controller_spec specifying that it should create and deliver an InviteMailer. spec/controllers/invites_controller_spec.rb

require 'spec_helper'

describe InvitesController do
  describe 'PUT #update' do
    let(:invite) { mock(:email => 'someone@someco.com', :save => true) }
    let(:invite_mailer) { mock }
    before do
      InviteMailer.should_receive(:invite_requested).with(invite).and_return(invite_mailer)
      invite_mailer.should_receive(:deliver)
      Invite.should_receive(:new).with('email' => 'someone@someco.com').and_return(invite)
      post :create, :invite => { :email => 'someone@someco.com' }
    end
    it { should redirect_to root_url }
    it { should set_the_flash.to("Thanks for your interest someone@someco.com.  You will hear from us soon.") }
  end
end

Of course we get an error uninitialized constant InviteMailer. We fix that by creating the mailer (it doesn’t do anything yet) app/mailers/invite_mailer.rb

class InviteMailer < ActionMailer::Base
end

The error changes

Failure/Error: InviteMailer.should_receive(:invite_requested).with(invite).and_return(invite_mailer)
  (<InviteMailer (class)>).invite_requested(#<RSpec::Mocks::Mock:0x82c96210 @name=nil>)
      expected: 1 time
      received: 0 times

We add the code to our controller to create and deliver the email in app/controllers/invites_controller.rb

class InvitesController < ApplicationController
  def create
    @invite = Invite.new(params[:invite])
    if @invite.save
      InviteMailer.invite_requested(@invite).deliver
      redirect_to root_path, :notice => "Thanks for your interest #{@invite.email}.  You will hear from us soon."
    else
      render :action => "new"
    end
  end
end

Are we done?

Checking the Cucumber Scenario…

The RSpec unit tests pass now but the Cucumber features are still failing.

When I request an invitation for "gullible@lemmings.com"            # features/step_definitions/invite_steps.rb:5
  undefined method `invite_requested' for InviteMailer:Class (NoMethodError)
  ./app/controllers/invites_controller.rb:10:in `create'
  (eval):2:in `send'
  (eval):2:in `click_button'
  ./features/step_definitions/invite_steps.rb:8:in `/^I request an invitation for "([^"]*)"$/'
  features/request_an_invitation.feature:8:in `When I request an invitation for "gullible@lemmings.com"'

Duh our InviteMailer doesn’t do anything.

Back down to the unit tests

We write our spec/mailers/invite_mailer_spec.rb. We need to include some EmailSpec modules so we have access to its matchers.

require 'spec_helper'

describe InviteMailer do
  include EmailSpec::Helpers
  include EmailSpec::Matchers

  describe '.invite_requested' do
    let(:invite) { Factory.build :invite, :email => 'someone@someco.com' }

    describe 'one email to one user' do
      subject { InviteMailer.invite_requested(invite) }
      it { should deliver_to     invite.email                                                  }
      it { should deliver_from   'alex@awesome-startup.com'                                    }
      it { should have_subject   "Invitation request for Awesome New Startup received"         }
      it { should have_body_text "Dear someone@someco.com,"                                    }
      it { should have_body_text "We have received your request"                               }
      it { should have_body_text "Please check back at http://awesome-site-staging.heroku.com" }
    end
  end
end

Of course it fails because we still haven’t implemented anything. Let’s add some code to app/mailers/invite_mailer.rb

class InviteMailer < ActionMailer::Base
  def invite_requested(invite)
    @invite = invite
    mail :to      => invite.email,
         :from    => 'alex@awesome-startup.com',
         :subject => 'Invitation request for Awesome New Startup received'
  end
end

and we can use haml to format the body in app/views/invite_mailer/invite_requested.text.haml

== Dear #{@invite.email},
We have received your request to be invited into our awesome site. We'll let you know as soon as its available.
Please check back at http://awesome-site-staging.heroku.com
You must be very excited!
Thanks
An Awesome New Startup

We run the rake one more time and …

We’re Done

Everything passes - the specs and the features!

$ rake
ruby -S rspec <a long list of _spec.rb files>
............

Finished in 2.64 seconds
12 examples, 0 failures

ruby -S bundle exec cucumber  --profile default
Using the default profile...
Feature: Build excitement for this vaporware
  In order to drum up interest
  As a user
  I will receive an exciting email

  Scenario: Someone requests an invitation and receives an email
    Given I am on the home page
    When I request an invitation for "gullible@lemmings.com"
    Then "gullible@lemmings.com" should receive 1 email
    And they open the email
    And they should see the email delivered from "alex@awesome-startup.com"
    And they should see "Invitation request for Awesome New Startup received" in the email subject
    And they should see "Dear gullible@lemmings.com," in the email body
    And they should see "We have received your request" in the email body
    And they should see "Please check back at http://awesome-site-staging.heroku.com" in the email body

1 scenario (1 passed)
9 steps (9 passed)
0m0.293s

I hope you’ll consider using the email_spec gem and BDD the next time you have to add email to your app.

Using BDD and the email_spec gem to implement Email

4 months ago | Alex Rothenberg: Common Sense Software

When implementing email functionality, the email_spec gem is something I’ve decided I can’t live without. It makes it so easy to write RSpec specs and Cucumber features around your email that you have no excuse not to. Today I’m going to go through an example how I recently used BDD to send an email in an app I was working on.

When I think about email and my Rails environments this is how I typically want them to behave.

  • test should not send emails and allow us to write specs or features against them
  • development should not send emails but provide a UI to view what would be sent on localhost
  • staging should not send emails but provide a UI to view what would be sent on a server
  • production should send real emails to real people

Last week I wrote about using letter opener to View Sent Email on a Server (without actually sending anything) describing how the letter_opener gem helps us in the development and staging environments. This article focuses on using email_spec gem in the test environment.

Our Sample Project

Let’s imagine we are working on a new startup in stealth mode. We want to generate buzz and prepare for a beta launch. We’re hiding the fact its all vaporware with a splashy homepage where people can request an invitation to the beta and records their email in our database. This is the same example as my article last week and there’s a live demo at http://awesome-site-staging.heroku.com/.

What we’ll do today is not just record the email address but also send a “thanks for your interest” email to the user. We’re going to do this in BDD fashion bouncing back and forth between Cucumber Scenarios and RSpec Unit Tests

  1. Writing a cucumber scenario

  2. While the scenario fails

    1. Write a failing rspec spec
    2. Write code to make it pass

The Cucumber scenario tells us what should be accomplished and when it fails it we use that to tell us what unit test we should write.

Adding the email_spec gem

We add it to our Gemfile

group :test do
  gem 'email_spec'
end

We bundle and use the email_spec generator to let it initialize itself.

$ bundle
$ rails g email_spec:steps

There’s also a manual step to get cucumber to load the email_spec gem. We need to create a file features/support/email_spec.rb

require 'email_spec/cucumber'

Our first Cucumber Scenario

We know that when a user requests an invitation they should get an email so we write that requirement as a Cucumber Scenario. features/request_an_invitation.feature

Feature: Build excitement for this vaporware
  In order to drum up interest
  As a user
  I will receive an exciting email when I request an invitation

  Scenario: Someone requests an invitation and receives an email
    Given I am on the home page
     When I request an invitation for "gullible@lemmings.com"
     Then "gullible@lemmings.com" should receive 1 email
      And they open the email
      And they should see the email delivered from "alex@awesome-startup.com"
      And they should see "Invitation request for Awesome New Startup received" in the email subject
      And they should see "Dear gullible@lemmings.com," in the email text part body
      And they should see "We have received your request " in the email text part body
      And they should see "Please check back at http://awesome-site-staging.heroku.com" in the email text part body

We run it and it tells us we have a couple of missing steps.

1 scenario (1 undefined)
9 steps (7 skipped, 2 undefined)
0m0.006s

You can implement step definitions for undefined steps with these snippets:

Given /^I am on the home page$/ do
  pending # express the regexp above with the code you wish you had
end

When /^I request an invitation for "([^"]*)"$/ do |arg1|
  pending # express the regexp above with the code you wish you had
end

Oh right, the training wheels came off in cucumber-rails v1.1.1 and we don’t have web_steps.rb anymore. Let’s write these steps using capybara in features/step_definitions/invite_steps.rb

Given /^I am on the home page$/ do
  visit root_path
end

When /^I request an invitation for "([^"]*)"$/ do |email|
  visit root_path
  fill_in 'email', :with => email
  click_button 'Request Invitation'
end

We run one more time and get a failure we expect. Its telling us we haven’t written any code to implement the scenario yet!

  Then "gullible@lemmings.com" should receive 1 email
  expected: 1
       got: 0 (using ==) (RSpec::Expectations::ExpectationNotMetError)
  ./features/step_definitions/email_steps.rb:52:in `/^(?:I|they|"([^"]*?)") should receive (an|no|\d+) emails?$/'
  features/request_an_invitation.feature:9:in `Then "gullible@lemmings.com" should receive 1 email'

Dropping into RSpec unit tests

Our failing feature tells us what we need to implement so we drop down to the unit test level and start implementing it with TDD. The feature tells us an email should be be generated so let’s go. We’ll add to our invites_controller_spec specifying that it should create and deliver an InviteMailer. spec/controllers/invites_controller_spec.rb

require 'spec_helper'

describe InvitesController do
  describe 'PUT #update' do
    let(:invite) { mock(:email => 'someone@someco.com', :save => true) }
    let(:invite_mailer) { mock }
    before do
      InviteMailer.should_receive(:invite_requested).with(invite).and_return(invite_mailer)
      invite_mailer.should_receive(:deliver)
      Invite.should_receive(:new).with('email' => 'someone@someco.com').and_return(invite)
      post :create, :invite => { :email => 'someone@someco.com' }
    end
    it { should redirect_to root_url }
    it { should set_the_flash.to("Thanks for your interest someone@someco.com.  You will hear from us soon.") }
  end
end

Of course we get an error uninitialized constant InviteMailer. We fix that by creating the mailer (it doesn’t do anything yet) app/mailers/invite_mailer.rb

class InviteMailer < ActionMailer::Base
end

The error changes

Failure/Error: InviteMailer.should_receive(:invite_requested).with(invite).and_return(invite_mailer)
  (<InviteMailer (class)>).invite_requested(#<RSpec::Mocks::Mock:0x82c96210 @name=nil>)
      expected: 1 time
      received: 0 times

We add the code to our controller to create and deliver the email in app/controllers/invites_controller.rb

class InvitesController < ApplicationController
  def create
    @invite = Invite.new(params[:invite])
    if @invite.save
      InviteMailer.invite_requested(@invite).deliver
      redirect_to root_path, :notice => "Thanks for your interest #{@invite.email}.  You will hear from us soon."
    else
      render :action => "new"
    end
  end
end

Are we done?

Checking the Cucumber Scenario…

The RSpec unit tests pass now but the Cucumber features are still failing.

When I request an invitation for "gullible@lemmings.com"            # features/step_definitions/invite_steps.rb:5
  undefined method `invite_requested' for InviteMailer:Class (NoMethodError)
  ./app/controllers/invites_controller.rb:10:in `create'
  (eval):2:in `send'
  (eval):2:in `click_button'
  ./features/step_definitions/invite_steps.rb:8:in `/^I request an invitation for "([^"]*)"$/'
  features/request_an_invitation.feature:8:in `When I request an invitation for "gullible@lemmings.com"'

Duh our InviteMailer doesn’t do anything.

Back down to the unit tests

We write our spec/mailers/invite_mailer_spec.rb. We need to include some EmailSpec modules so we have access to its matchers.

require 'spec_helper'

describe InviteMailer do
  include EmailSpec::Helpers
  include EmailSpec::Matchers

  describe '.invite_requested' do
    let(:invite) { Factory.build :invite, :email => 'someone@someco.com' }

    describe 'one email to one user' do
      subject { InviteMailer.invite_requested(invite) }
      it { should deliver_to     invite.email                                                  }
      it { should deliver_from   'alex@awesome-startup.com'                                    }
      it { should have_subject   "Invitation request for Awesome New Startup received"         }
      it { should have_body_text "Dear someone@someco.com,"                                    }
      it { should have_body_text "We have received your request"                               }
      it { should have_body_text "Please check back at http://awesome-site-staging.heroku.com" }
    end
  end
end

Of course it fails because we still haven’t implemented anything. Let’s add some code to app/mailers/invite_mailer.rb

class InviteMailer < ActionMailer::Base
  def invite_requested(invite)
    @invite = invite
    mail :to      => invite.email,
         :from    => 'alex@awesome-startup.com',
         :subject => 'Invitation request for Awesome New Startup received'
  end
end

and we can use haml to format the body in app/views/invite_mailer/invite_requested.text.haml

== Dear #{@invite.email},
We have received your request to be invited into our awesome site. We'll let you know as soon as its available.
Please check back at http://awesome-site-staging.heroku.com
You must be very excited!
Thanks
An Awesome New Startup

We run the rake one more time and …

We’re Done

Everything passes - the specs and the features!

$ rake
ruby -S rspec <a long list of _spec.rb files>
............

Finished in 2.64 seconds
12 examples, 0 failures

ruby -S bundle exec cucumber  --profile default
Using the default profile...
Feature: Build excitement for this vaporware
  In order to drum up interest
  As a user
  I will receive an exciting email

  Scenario: Someone requests an invitation and receives an email
    Given I am on the home page
    When I request an invitation for "gullible@lemmings.com"
    Then "gullible@lemmings.com" should receive 1 email
    And they open the email
    And they should see the email delivered from "alex@awesome-startup.com"
    And they should see "Invitation request for Awesome New Startup received" in the email subject
    And they should see "Dear gullible@lemmings.com," in the email body
    And they should see "We have received your request" in the email body
    And they should see "Please check back at http://awesome-site-staging.heroku.com" in the email body

1 scenario (1 passed)
9 steps (9 passed)
0m0.293s

I hope you’ll consider using the email_spec gem and BDD the next time you have to add email to your app.

Using Letter Opener to View Sent Email on a Server (without actually sending anything)

4 months ago | Alex Rothenberg: Common Sense Software

When developing email functionality you don’t want to send real emails to real people before in production. At the same time you need to send them to ensure they are formatted correctly and contain the proper information. You can (and should) write integration tests to verify this but that helps developers gain confidence, what can we do to show non-technical stakeholders that it all works?

Today I’m going to show you how to use Ryan Bates’ letter_opener gem to let you preview your emails without actually sending them. Ryan of course does the always awesome RailsCasts.

Let’s think about the different Rails environments and how we want them to behave with email.

  • test should not send emails and allow us to write specs or features against them
  • development should not send emails but provide a UI to view what would be sent
  • staging should not send emails but provide a UI to view what would be sent
  • production should send real emails to real people

Looking at this, Rails works really well for test with ActionMailer’s :test delivery method that stores the emails in the ActionMailer::Base.deliveries array so you can then use email_spec to write your specs. Production is also covered as long as you give it your mail server configuration. That leaves development and staging which look identical but we’ll see that they are slightly different. I’ll spend the rest of this article talking about how letter_opener lets us do what we want in these environments.

An example

Let’s imagine we are working on a new startup in stealth mode. We want to generate buzz and prepare for a beta launch. We’re hiding the fact its all vaporware with a splashy homepage where people can request an invitation to the beta and it sends a “thanks for your interest” email. Lastly, we want to test it in development and staging.

Here’s a live demo at http://awesome-site-staging.heroku.com/ if you want to dive in and start clicking.

After you request an invitation it shows a page like this

Vaporware Homepage

Logfile testing (we can do better)

When you fill in your email and click the Request Invitation button, our controller uses a mailer to create and deliver the email.

class InvitesController < ApplicationController
  def create
    @invite = Invite.new(params[:invite])
    @invite.save
    InviteMailer.invite_requested(@invite).deliver
    redirect_to root_path, :notice => "Thanks for your interest #{@invite.email}.  You will hear from us soon."
  end
end

The only way to tell whether the email worked is to scroll through the development.log until you see something like this

Sent mail to alex@alexrothenberg.com (20ms)
Date: Fri, 21 Oct 2011 15:09:16 -0400
From: admin@newstartup.com
To: alex@alexrothenberg.com
Message-ID: <4ea1c35cc3d5b_6693830916bc43754@Alex-Rothenbergs-MacBook-Pro.local.mail>
Subject: Invite requested
...
We have received your request to be invited into our awesome site. We'll let you know as soon as its available.
Please check back at http://awesome-site.heroku.com
You must be very excited!
Thanks
An Awesome New Startup

Ok if you’re a developer and you enjoy reading log files but letter_opener lets us do better.

Using letter_opener in development

Letter_opener provides us with a UI so we can view the emails right in our browser. It’s super easy to add this gem and I’ll just copy the instructions from its README

Preview email in the browser instead of sending it. This means you do not need to set up email delivery in your development environment, and you no longer need to worry about accidentally sending a test email to someone else’s address

Rails Setup

First add the gem to your development environment and run the bundle command to install it.

gem "letter_opener", :group => :development

Then set the delivery method in config/environments/development.rb

config.action_mailer.delivery_method = :letter_opener

Now any email will pop up in your browser instead of being sent. The messages are stored in tmp/letter_opener.

Once we’ve done that what happens when we use the site to request an invitation? A new tab opens up with the email right there. Now as a user we can tell it’s correct and any non-technical people on the team can feel their confidence rise.

Previewing an Email with Letter Opener in Development

How does letter_opener actually work?

Rails goes through the standard flow to create the mail object and when it’s ready to deliver the message it calls letter_opener’s deliver! method because we registered letter_opener as the action_mailer.delivery_method. Letter_opener saves the email to your file system as an html file then uses launchy to open it in a browser using the file:// protocol.

There will be a couple of problems once we move onto a server which brings us to staging.

Using letter_opener on staging

If you’re like me you probably have a staging environment where you or your stakeholders can validate your app before releasing it to production and your end users. There are two aspects of this environment that wont work with letter_opener the way it did in development

  • We need to use http:// not file:// to preview the emails because the browser is not on the same file system where the emails are written
  • We may not be able to write to the file system. For example if we have deployed to heroku.

I had to make some changes to letter_opener to support this kind of server environment. The fork is available at https://github.com/alexrothenberg/letter_opener/tree/on_a_server and I’ll update the article if my pull requests are merged back in.

We need to make a few changes to our application.

1 - Update our Gemfile to use the fork from github

  gem 'letter_opener',  :git => "git://github.com/alexrothenberg/letter_opener.git", :branch => "on_a_server"
  

2 - Add a debugging UI link so users can get to the “preview emails” page in something like layouts/application.html.haml

  = link_to 'Preview Emails', letter_opener_letters_path if Rails.env.staging?
  

3 - If you cannot write to the filesystem let letter_opener know in your config/environments/staging.rb

  config.action_mailer.delivery_method = :letter_opener
  LetterOpener.cannot_write_to_file_system!
  

Now we can see it all in action.

First, we request an invite.

Previewing an Email with Letter Opener in Development

Then, we click the link “view the Emails that users would have received” link at the bottom and see an “inbox” of everything the app sent.

Previewing an Email with Letter Opener in Development

Fincally clicking one message lets us preview it just as we did in devellopment

Previewing an Email with Letter Opener in Development

We are able to run on heroku without writing to the file system by using the FakeFS gem which simulates the filesystem in memory. FakeFS is designed to be used for testing and one caveat to be aware of is that your old emails will disappear if heroku recycles your dyno due to inactivity.

I hope you think letter_opener is useful and give it a try the next time you need to send email. Remember here’s a live demo at http://awesome-site-staging.heroku.com/ of the site we’ve been talking about here.

Using Letter Opener to View Sent Email on a Server (without actually sending anything)

4 months ago | Alex Rothenberg: Common Sense Software

When developing email functionality you don’t want to send real emails to real people before in production. At the same time you need to send them to ensure they are formatted correctly and contain the proper information. You can (and should) write integration tests to verify this but that helps developers gain confidence, what can we do to show non-technical stakeholders that it all works?

Today I’m going to show you how to use Ryan Bates’ letter_opener gem to let you preview your emails without actually sending them. Ryan of course does the always awesome RailsCasts.

Let’s think about the different Rails environments and how we want them to behave with email.

  • test should not send emails and allow us to write specs or features against them
  • development should not send emails but provide a UI to view what would be sent
  • staging should not send emails but provide a UI to view what would be sent
  • production should send real emails to real people

Looking at this, Rails works really well for test with ActionMailer’s :test delivery method that stores the emails in the ActionMailer::Base.deliveries array so you can then use email_spec to write your specs. Production is also covered as long as you give it your mail server configuration. That leaves development and staging which look identical but we’ll see that they are slightly different. I’ll spend the rest of this article talking about how letter_opener lets us do what we want in these environments.

An example

Let’s imagine we are working on a new startup in stealth mode. We want to generate buzz and prepare for a beta launch. We’re hiding the fact its all vaporware with a splashy homepage where people can request an invitation to the beta and it sends a “thanks for your interest” email. Lastly, we want to test it in development and staging.

Here’s a live demo at http://awesome-site-staging.heroku.com/ if you want to dive in and start clicking.

After you request an invitation it shows a page like this

Vaporware Homepage

Logfile testing (we can do better)

When you fill in your email and click the Request Invitation button, our controller uses a mailer to create and deliver the email.

class InvitesController < ApplicationController
  def create
    @invite = Invite.new(params[:invite])
    @invite.save
    InviteMailer.invite_requested(@invite).deliver
    redirect_to root_path, :notice => "Thanks for your interest #{@invite.email}.  You will hear from us soon."
  end
end

The only way to tell whether the email worked is to scroll through the development.log until you see something like this

Sent mail to alex@alexrothenberg.com (20ms)
Date: Fri, 21 Oct 2011 15:09:16 -0400
From: admin@newstartup.com
To: alex@alexrothenberg.com
Message-ID: <4ea1c35cc3d5b_6693830916bc43754@Alex-Rothenbergs-MacBook-Pro.local.mail>
Subject: Invite requested
...
We have received your request to be invited into our awesome site. We'll let you know as soon as its available.
Please check back at http://awesome-site.heroku.com
You must be very excited!
Thanks
An Awesome New Startup

Ok if you’re a developer and you enjoy reading log files but letter_opener lets us do better.

Using letter_opener in development

Letter_opener provides us with a UI so we can view the emails right in our browser. It’s super easy to add this gem and I’ll just copy the instructions from its README

Preview email in the browser instead of sending it. This means you do not need to set up email delivery in your development environment, and you no longer need to worry about accidentally sending a test email to someone else’s address

Rails Setup

First add the gem to your development environment and run the bundle command to install it.

gem "letter_opener", :group => :development

Then set the delivery method in config/environments/development.rb

config.action_mailer.delivery_method = :letter_opener

Now any email will pop up in your browser instead of being sent. The messages are stored in tmp/letter_opener.

Once we’ve done that what happens when we use the site to request an invitation? A new tab opens up with the email right there. Now as a user we can tell it’s correct and any non-technical people on the team can feel their confidence rise.

Previewing an Email with Letter Opener in Development

How does letter_opener actually work?

Rails goes through the standard flow to create the mail object and when it’s ready to deliver the message it calls letter_opener’s deliver! method because we registered letter_opener as the action_mailer.delivery_method. Letter_opener saves the email to your file system as an html file then uses launchy to open it in a browser using the file:// protocol.

There will be a couple of problems once we move onto a server which brings us to staging.

Using letter_opener on staging

If you’re like me you probably have a staging environment where you or your stakeholders can validate your app before releasing it to production and your end users. There are two aspects of this environment that wont work with letter_opener the way it did in development

  • We need to use http:// not file:// to preview the emails because the browser is not on the same file system where the emails are written
  • We may not be able to write to the file system. For example if we have deployed to heroku.

I had to make some changes to letter_opener to support this kind of server environment. The fork is available at https://github.com/alexrothenberg/letter_opener/tree/on_a_server and I’ll update the article if my pull requests are merged back in.

We need to make a few changes to our application.

1 - Update our Gemfile to use the fork from github

  gem 'letter_opener',  :git => "git://github.com/alexrothenberg/letter_opener.git", :branch => "on_a_server"
  

2 - Add a debugging UI link so users can get to the “preview emails” page in something like layouts/application.html.haml

  = link_to 'Preview Emails', letter_opener_letters_path if Rails.env.staging?
  

3 - If you cannot write to the filesystem let letter_opener know in your config/environments/staging.rb

  config.action_mailer.delivery_method = :letter_opener
  LetterOpener.cannot_write_to_file_system!
  

Now we can see it all in action.

First, we request an invite.

Previewing an Email with Letter Opener in Development

Then, we click the link “view the Emails that users would have received” link at the bottom and see an “inbox” of everything the app sent.

Previewing an Email with Letter Opener in Development

Fincally clicking one message lets us preview it just as we did in devellopment

Previewing an Email with Letter Opener in Development

We are able to run on heroku without writing to the file system by using the FakeFS gem which simulates the filesystem in memory. FakeFS is designed to be used for testing and one caveat to be aware of is that your old emails will disappear if heroku recycles your dyno due to inactivity.

I hope you think letter_opener is useful and give it a try the next time you need to send email. Remember here’s a live demo at http://awesome-site-staging.heroku.com/ of the site we’ve been talking about here.

Chaining ActiveRecord Scopes from Different Models

4 months ago | Alex Rothenberg: Common Sense Software

You probably know you can chain ActiveRecord scopes together and it combines it all together into one sql statement, but did you know that you can also use scopes from associated models in your chain?

Let me show you with an example. Let’s say we have a discussion site where users can post comments with two simple models:

class User < ActiveRecord::Base
  has_many :comments
end

class Comment < ActiveRecord::Base
  belongs_to :user
end

We could display a list of all users with User.all or even display them alphabetically with User.order(:name) (yes I’d create a scope if I were doing this for real).

What if I wanted to display the users sorted so that the ones with the most recent comment was first?

User.includes(:comments).merge(Comment.order('comments.created_at desc'))

There’s a lot going on there, let’s break it down.

  • User.includes(:comments) - tells active record to query both the users and comments tables
  • Comment.order('comments.created_at desc') - sorts the results by the date of the comment (we need to specify the table and column name since created_at is also a column on the users table)
  • merge(Comment.XXX) - lets us use a scope from the Comment model even though we’re dealing with Users

When ActiveRecord and ActiveRelation take all this and convert it into a sql statement it will join the users and comments tables and order by the comments.created_at column. Here’s the sql I actually get in irb (boy am I glad I didn’t have to type that sql myself!).

> User.includes(:comments).merge(Comment.order('comments.created_at desc'))
  SQL (0.4ms)  SELECT "users"."id" AS t0_r0, "users"."name" AS t0_r1, "users"."created_at" AS t0_r2,
                      "users"."updated_at" AS t0_r3, "comments"."id" AS t1_r0, "comments"."user_id" AS t1_r1,
                      "comments"."body" AS t1_r2, "comments"."created_at" AS t1_r3, "comments"."updated_at" AS t1_r4
               FROM "users"
               LEFT OUTER JOIN "comments" ON "comments"."user_id" = "users"."id"
               ORDER BY comments.created_at desc

Adding Scopes to make it usable

It works to type all that but in a real application you’d add scopes to make it easier to work with. Let’s do that!

class User < ActiveRecord::Base
  has_many :comments
  scope :by_most_recent_comment, includes(:comments).merge(Comment.most_recent_first)
  scope :with_recent_comments, includes(:comments).merge(Comment.recently(1.month.ago))
end

class Comment < ActiveRecord::Base
  belongs_to :user
  scope :most_recent_first, order('comments.created_at desc')
  scope :recently, lambda { |date| where('comments.created_at >= ?', date) }
end

Now we can write some nice simple scopes like

  • User.by_most_recent_comment to get all users sorted so the ones with recent comments are at the top
  • User.with_recent_comments to get all users who have commented in the past month
  • User.with_recent_comments.by_most_recent_comment to get users who have commented in the past month sorted by the date of their most recent comment.

Happy scoping!

Chaining ActiveRecord Scopes from Different Models

4 months ago | Alex Rothenberg: Common Sense Software

You probably know you can chain ActiveRecord scopes together and it combines it all together into one sql statement, but did you know that you can also use scopes from associated models in your chain?

Let me show you with an example. Let’s say we have a discussion site where users can post comments with two simple models:

class User < ActiveRecord::Base
  has_many :comments
end

class Comment < ActiveRecord::Base
  belongs_to :user
end

We could display a list of all users with User.all or even display them alphabetically with User.order(:name) (yes I’d create a scope if I were doing this for real).

What if I wanted to display the users sorted so that the ones with the most recent comment was first?

User.includes(:comments).merge(Comment.order('comments.created_at desc'))

There’s a lot going on there, let’s break it down.

  • User.includes(:comments) - tells active record to query both the users and comments tables
  • Comment.order('comments.created_at desc') - sorts the results by the date of the comment (we need to specify the table and column name since created_at is also a column on the users table)
  • merge(Comment.XXX) - lets us use a scope from the Comment model even though we’re dealing with Users

When ActiveRecord and ActiveRelation take all this and convert it into a sql statement it will join the users and comments tables and order by the comments.created_at column. Here’s the sql I actually get in irb (boy am I glad I didn’t have to type that sql myself!).

> User.includes(:comments).merge(Comment.order('comments.created_at desc'))
  SQL (0.4ms)  SELECT "users"."id" AS t0_r0, "users"."name" AS t0_r1, "users"."created_at" AS t0_r2,
                      "users"."updated_at" AS t0_r3, "comments"."id" AS t1_r0, "comments"."user_id" AS t1_r1,
                      "comments"."body" AS t1_r2, "comments"."created_at" AS t1_r3, "comments"."updated_at" AS t1_r4
               FROM "users"
               LEFT OUTER JOIN "comments" ON "comments"."user_id" = "users"."id"
               ORDER BY comments.created_at desc

Adding Scopes to make it usable

It works to type all that but in a real application you’d add scopes to make it easier to work with. Let’s do that!

class User < ActiveRecord::Base
  has_many :comments
  scope :by_most_recent_comment, includes(:comments).merge(Comment.most_recent_first)
  scope :with_recent_comments, includes(:comments).merge(Comment.recently(1.month.ago))
end

class Comment < ActiveRecord::Base
  belongs_to :user
  scope :most_recent_first, order('comments.created_at desc')
  scope :recently, lambda { |date| where('comments.created_at >= ?', date) }
end

Now we can write some nice simple scopes like

  • User.by_most_recent_comment to get all users sorted so the ones with recent comments are at the top
  • User.with_recent_comments to get all users who have commented in the past month
  • User.with_recent_comments.by_most_recent_comment to get users who have commented in the past month sorted by the date of their most recent comment.

Happy scoping!

Ammeter: The Way to Write Specs for Your Rails Generators

5 months ago | Alex Rothenberg: Common Sense Software

Generators got a complete makeover with Rails 3 making them much easier to write but they’ve been very hard to test if you’re using RSpec. That’s changed now with the Ammeter Gem which lets you write RSpec specs for your generators.

Who writes generators?

Unless you’ve writing a gem you probably haven’t created a generator, but I bet you’re using one someone else created. If you’ve ever typed

  • “rails g rspec:install” and a spec director appeared
  • “rails g cucumber:install” and gotten a features directory
  • “rails g model post title:string body:text” and gotten specs for your model
  • “rails g model post title:string body:text” and gotten mongoid models insead active record ones

There are a number of resources for writing generators using thor including the generators guide or railscast #218 and I’m not going to go into that here. If you’re using TestUnit, like the rails core team, you can use Generators::TestCase which is part of Rails - Devise has some good examples. For those of us using RSpec we can now use Ammeter.

Writing Specs with Ammeter

First you need to tell your gem to use ammeter by 1) adding it to our bundle and 2) making it accessible to our specs.

  # <YOUR_GEM_NAME>.gemspec
  s.add_development_dependency 'ammeter'
  # spec_helper.rb
  require 'ammeter/init'

Then we specify the behavior. We’ll look at an example using Mongoid’s config generator and its spec config_generator_spec. The generator’s usage is rails generate mongoid:config [DATABASE_NAME] [options].

# spec/generators/mongoid/config/config_generator_spec.rb
require 'spec_helper'

# Generators are not automatically loaded by Rails
require 'rails/generators/mongoid/config/config_generator'

class Rails::Application; end
class MyApp::Application < Rails::Application; end

describe Mongoid::Generators::ConfigGenerator do
  # Tell the generator where to put its output (what it thinks of as Rails.root)
  destination File.expand_path("../../../../../../tmp", __FILE__)
  before { prepare_destination }

  describe 'no arguments' do
    before { run_generator  }
    describe 'config/mongoid.yml' do
      subject { file('config/mongoid.yml') }
      it { should exist }
      it { should contain "database: my_app_development" }
    end
  end

  describe 'specifying database name' do
    before { run_generator %w(my_database) }
    describe 'config/mongoid.yml' do
      subject { file('config/mongoid.yml') }
      it { should exist }
      it { should contain "database: my_database_development" }
    end
  end
end

There’s some boilerplate setup you’ll need at the top of your spec:

  • Since this spec file is in spec/generators it automatically uses ammeter
  • Generators are not automatically loaded by Rails’ const_missing so we need to require it explicitly
  • destination tells the generator where to put its output (we add enough “../”s to get us out of the spec directory)
  • before { prepare_destination } clears the destination so each spec starts fresh (similar to active record rollback)

Now for the behavior:

  • run_generator runs the generator (optionally letting you pass in arguments)
  • file gives you access to a generated file
  • it should exist makes sure the file was generated
  • it should contain looks inside a generated file for a string or regex

When you’re generating migrations it is somewhat tricky because migration file names contain a timestamp. We need another example and will use acts_as_taggable-on’s migration generator We could write a spec for this as

# spec/generators/acts_as_taggable_on/migration/migration_generator_spec.rb
require 'spec_helper'

# Generators are not automatically loaded by Rails
require 'generators/acts_as_taggable_on/migration/migration_generator'

describe ActsAsTaggableOn::MigrationGenerator do
  # Tell the generator where to put its output (what it thinks of as Rails.root)
  destination File.expand_path("../../../../../tmp", __FILE__)

  before do
    prepare_destination
    Rails::Generators.options[:rails][:orm] = :active_record
  end
  describe 'no arguments' do
    before { run_generator  }

    describe 'db/migrate/acts_as_taggable_on_migration.rb' do
      subject { file('db/migrate/acts_as_taggable_on_migration.rb') }
      it { should be_a_migration }
    end
  end
end

You can see much of the same setup and then the new matcher

  • it { should be_a_migration } which adds a timestamp to the filename

Why ‘Ammeter’?

An Ammeter is a measuring instrument used to measure the electric current in a circuit. Generators produce electricity and your specs measure your generators … cute huh :)

Feedback Welcome

Try Ammeter for your generators, I hope you find it useful. If it doesn’t meet your needs fork away - Ammeter is on github. I welcome any feedback, issues or pull requests.

Ammeter: The Way to Write Specs for Your Rails Generators

5 months ago | Alex Rothenberg: Common Sense Software

Generators got a complete makeover with Rails 3 making them much easier to write but they’ve been very hard to test if you’re using RSpec. That’s changed now with the Ammeter Gem which lets you write RSpec specs for your generators.

Who writes generators?

Unless you’ve writing a gem you probably haven’t created a generator, but I bet you’re using one someone else created. If you’ve ever typed

  • “rails g rspec:install” and a spec director appeared
  • “rails g cucumber:install” and gotten a features directory
  • “rails g model post title:string body:text” and gotten specs for your model
  • “rails g model post title:string body:text” and gotten mongoid models insead active record ones

There are a number of resources for writing generators using thor including the generators guide or railscast #218 and I’m not going to go into that here. If you’re using TestUnit, like the rails core team, you can use Generators::TestCase which is part of Rails - Devise has some good examples. For those of us using RSpec we can now use Ammeter.

Writing Specs with Ammeter

First you need to tell your gem to use ammeter by 1) adding it to our bundle and 2) making it accessible to our specs.

  # <YOUR_GEM_NAME>.gemspec
  s.add_development_dependency 'ammeter'
  # spec_helper.rb
  require 'ammeter/init'

Then we specify the behavior. We’ll look at an example using Mongoid’s config generator and its spec config_generator_spec. The generator’s usage is rails generate mongoid:config [DATABASE_NAME] [options].

# spec/generators/mongoid/config/config_generator_spec.rb
require 'spec_helper'

# Generators are not automatically loaded by Rails
require 'rails/generators/mongoid/config/config_generator'

class Rails::Application; end
class MyApp::Application < Rails::Application; end

describe Mongoid::Generators::ConfigGenerator do
  # Tell the generator where to put its output (what it thinks of as Rails.root)
  destination File.expand_path("../../../../../../tmp", __FILE__)
  before { prepare_destination }

  describe 'no arguments' do
    before { run_generator  }
    describe 'config/mongoid.yml' do
      subject { file('config/mongoid.yml') }
      it { should exist }
      it { should contain "database: my_app_development" }
    end
  end

  describe 'specifying database name' do
    before { run_generator %w(my_database) }
    describe 'config/mongoid.yml' do
      subject { file('config/mongoid.yml') }
      it { should exist }
      it { should contain "database: my_database_development" }
    end
  end
end

There’s some boilerplate setup you’ll need at the top of your spec:

  • Since this spec file is in spec/generators it automatically uses ammeter
  • Generators are not automatically loaded by Rails’ const_missing so we need to require it explicitly
  • destination tells the generator where to put its output (we add enough “../”s to get us out of the spec directory)
  • before { prepare_destination } clears the destination so each spec starts fresh (similar to active record rollback)

Now for the behavior:

  • run_generator runs the generator (optionally letting you pass in arguments)
  • file gives you access to a generated file
  • it should exist makes sure the file was generated
  • it should contain looks inside a generated file for a string or regex

When you’re generating migrations it is somewhat tricky because migration file names contain a timestamp. We need another example and will use acts_as_taggable-on’s migration generator We could write a spec for this as

# spec/generators/acts_as_taggable_on/migration/migration_generator_spec.rb
require 'spec_helper'

# Generators are not automatically loaded by Rails
require 'generators/acts_as_taggable_on/migration/migration_generator'

describe ActsAsTaggableOn::MigrationGenerator do
  # Tell the generator where to put its output (what it thinks of as Rails.root)
  destination File.expand_path("../../../../../tmp", __FILE__)

  before do
    prepare_destination
    Rails::Generators.options[:rails][:orm] = :active_record
  end
  describe 'no arguments' do
    before { run_generator  }

    describe 'db/migrate/acts_as_taggable_on_migration.rb' do
      subject { file('db/migrate/acts_as_taggable_on_migration.rb') }
      it { should be_a_migration }
    end
  end
end

You can see much of the same setup and then the new matcher

  • it { should be_a_migration } which adds a timestamp to the filename

Why ‘Ammeter’?

An Ammeter is a measuring instrument used to measure the electric current in a circuit. Generators produce electricity and your specs measure your generators … cute huh :)

Feedback Welcome

Try Ammeter for your generators, I hope you find it useful. If it doesn’t meet your needs fork away - Ammeter is on github. I welcome any feedback, issues or pull requests.

Upgrading An Old Rails Rails 2.1.1 to 3.1 on Heroku

5 months ago | Alex Rothenberg: Common Sense Software

I recently had to upgrade an old app and it went pretty smoothly so I thought I’d share how I did it.

I thought about two approaches to this upgrade

  1. I could slowly add modernity to the existing app by adding bundler, upgrading rails, upgrading rspec, etc.
  2. I could create a new Rails 3.1 project and copy the code and specs into this new project.

I downloaded the old code and immediately remembered the pain of getting an app up and running without bundler. This convinced me that #1 would be harder than it needed to be so I decided to go with approach #2. It worked out pretty well so I’m going to share exactly what I did.

Create a new Rails 3.1 project

$ rails new waywework
      create
      create  README
      create  Rakefile
      # and a lot more...
         run  bundle install
Fetching source index for http://rubygems.org/
Using rake (0.9.2)
# and a lot more...
Your bundle is complete! Use `bundle show [gemname]` to see where a bundled gem is installed.

Add gems from my app

Because the old app comes from the days before bundler I had to look in config/environment.rb for lines like config.gem 'atom' and guess what else I needed (I knew I was using rspec).

#Gemfile
gem 'atom'
group :development, :test do
  gem "rspec-rails"
  gem "factory_girl_rails"
  gem "haml-rails"
  gem 'faker'
end

Now we can make sure our empty app works (even though it does nothing)

$ cd waywework
$ rails g rspec:install
      create  .rspec
      create  spec
      create  spec/spec_helper.rb
$ rake db:migrate
$ rake
No examples matching ./spec/**/*_spec.rb could be found

Add our existing code and specs

$ cp -r ~/old_waywework/app/controllers/* app/controllers
$ cp -r ~/old_waywework/app/models/* app/models
$ cp -r ~/old_waywework/app/helpers/* app/helpers
$ cp -r ~/old_waywework/app/views/* app/views
$ cp -r ~/old_waywework/db/migrate db
$ mv spec/spec_helper.rb new_spec_helper.rb
$ cp -r ~/old_waywework/spec/* spec
$ mv new_spec_helper.rb spec/spec_helper.rb

Now when we try to run the specs it tells us we have pending migrations so we run them

$ rake
You have 5 pending migrations:
  20081022162743 CreateFeeds
  20081022162832 CreatePosts
  20081031025747 IndexPublishedInPosts
  20081031143453 IndexFeedAuthor
  20081103143931 PublishedAsDatetime
$ rake db:migrate
==  CreateFeeds: migrating ====================================================
-- create_table(:feeds)
   -> 0.0025s
==  CreateFeeds: migrated (0.0026s) ===========================================

==  CreatePosts: migrating ====================================================
-- create_table(:posts)
   -> 0.0021s
==  CreatePosts: migrated (0.0023s) ===========================================

==  IndexPublishedInPosts: migrating ==========================================
-- add_index(:posts, [:published, :feed_id])
   -> 0.0015s
==  IndexPublishedInPosts: migrated (0.0017s) =================================

==  IndexFeedAuthor: migrating ================================================
-- add_index(:feeds, :author)
   -> 0.0007s
==  IndexFeedAuthor: migrated (0.0008s) =======================================

==  PublishedAsDatetime: migrating ============================================
-- change_column(:posts, :published, :datetime)
   -> 0.0086s
-- add_column(:posts, :updated, :datetime)
   -> 0.0006s
==  PublishedAsDatetime: migrated (0.0094s) ===================================
$ rake
...LOTS OF ERRORS...
activerecord-3.1.0/lib/active_record/base.rb:1083:
    in `method_missing': undefined method `named_scope' for #<Class:0x103875b28> (NoMethodError)

Now we’re getting some errors and are ready to get to work.

Update our code

Of course it doesn’t just work and we need to make a number of changes. The guides include an upgrade process section and the rails_upgrade plugin can help identify problems. Here are the ones I found.

Changing named_scope to scope

With Rails 3 the syntax for scopes changed from named_scope to scope. So we search-and-replace in our models. Once we do this the specs run, but many are failing. A lot are failing because the routes.rb syntax changed in Rails 3.

RAILS_ROOT => Rails.root

The constant RAILS_ROOT no longer exists so we need to search-and-replace that with Rails.root.

Routes.rb

# OLD config/routes.rb
ActionController::Routing::Routes.draw do |map|
  map.posts_by_author 'author/:id', :controller=>'posts', :action=>'by_author'
  map.posts_by_month 'month/:year/:month', :controller=>'posts', :action=>'by_month'
  map.atom_feed '/atom', :controller => "posts", :action=>'index', :format=>'atom'

  map.resources :feeds
  map.root :controller => "posts"
end

We change it to:

# NEW config/routes.rb
WayWeWork::Application.routes.draw do
  match 'author/:id' =>'posts#by_author', :as => :posts_by_author
  match 'month/:year/:month' =>'posts#by_month', :as => :posts_by_month
  match '/atom' => 'posts#index', :as => :atom_feed, :format => :atom

  resources :feeds

  root :to => 'posts#index'
end

RSpec Routing Specs

We have a bunch of routing specs that are failing because RSpec changed the syntax around routing specs.

# OLD spec/controllers/feeds_routing_spec.rb
it "should map #index" do
  route_for(:controller => "feeds", :action => "index").should == "/feeds"
end
it "should generate params for #index" do
  params_from(:get, "/feeds").should == {:controller => "feeds", :action => "index"}
end

The routing spec must be in folder called routing

# NEW spec/feeds_routing_spec.rb
it "should generate params for #index" do
  get("/feeds").should route_to('feeds#index')
end

RSpec stub!

Another syntax change in RSpec from using stubs to stub!

# OLD
it 'should ...' do
  #...
  feed.stubs(:puts)
  feed.get_latest
end

We get an error undefined method 'stubs' for \#<Feed:0x105d572e8>

# NEW
it 'should ...' do
  #...
  feed.stub!(:puts)
  feed.get_latest
end

View Specs

Hmm I’m surprised I had written view specs. Nowadays I would write cucumber features and never write view specs. That being said there were a few things I needed to change to get the view specs passing, all related to RSpec syntax changes.

I was passing @ variables to the view with assigns[:feed] = @feed = stub_model(Feed) and today you need to do that with assign(:feed, @feed = stub_model(Feed). Also when you call render "/feeds/edit.html.erb" it used to work but now tries to render a partial. Plain old render will work as long as you’ve put the filename in your describe like describe "/feeds/edit.html.erb" do

Deploying to Heroku

The old app was deployed to a VPS using Capistrano. Today I prefer to use Heroku and there were a few changes I had to make to get it working there.

  • Add pg to my Gemfile to support Postgresql
  • Configure assets (since we cannot write to the filesystem) ** Added config.assets.compile = true in my production.rb ** Added therubyracer to my Gemfile
  • Delete my old Capfile

Now I was able to deploy to heroku and the new app is up and running. All this done in less than a day!

Upgrading An Old Rails Rails 2.1.1 to 3.1 on Heroku

5 months ago | Alex Rothenberg: Common Sense Software

I recently had to upgrade an old app and it went pretty smoothly so I thought I’d share how I did it.

I thought about two approaches to this upgrade

  1. I could slowly add modernity to the existing app by adding bundler, upgrading rails, upgrading rspec, etc.
  2. I could create a new Rails 3.1 project and copy the code and specs into this new project.

I downloaded the old code and immediately remembered the pain of getting an app up and running without bundler. This convinced me that #1 would be harder than it needed to be so I decided to go with approach #2. It worked out pretty well so I’m going to share exactly what I did.

Create a new Rails 3.1 project

$ rails new waywework
      create
      create  README
      create  Rakefile
      # and a lot more...
         run  bundle install
Fetching source index for http://rubygems.org/
Using rake (0.9.2)
# and a lot more...
Your bundle is complete! Use `bundle show [gemname]` to see where a bundled gem is installed.

Add gems from my app

Because the old app comes from the days before bundler I had to look in config/environment.rb for lines like config.gem 'atom' and guess what else I needed (I knew I was using rspec).

#Gemfile
gem 'atom'
group :development, :test do
  gem "rspec-rails"
  gem "factory_girl_rails"
  gem "haml-rails"
  gem 'faker'
end

Now we can make sure our empty app works (even though it does nothing)

$ cd waywework
$ rails g rspec:install
      create  .rspec
      create  spec
      create  spec/spec_helper.rb
$ rake db:migrate
$ rake
No examples matching ./spec/**/*_spec.rb could be found

Add our existing code and specs

$ cp -r ~/old_waywework/app/controllers/* app/controllers
$ cp -r ~/old_waywework/app/models/* app/models
$ cp -r ~/old_waywework/app/helpers/* app/helpers
$ cp -r ~/old_waywework/app/views/* app/views
$ cp -r ~/old_waywework/db/migrate db
$ mv spec/spec_helper.rb new_spec_helper.rb
$ cp -r ~/old_waywework/spec/* spec
$ mv new_spec_helper.rb spec/spec_helper.rb

Now when we try to run the specs it tells us we have pending migrations so we run them

$ rake
You have 5 pending migrations:
  20081022162743 CreateFeeds
  20081022162832 CreatePosts
  20081031025747 IndexPublishedInPosts
  20081031143453 IndexFeedAuthor
  20081103143931 PublishedAsDatetime
$ rake db:migrate
==  CreateFeeds: migrating ====================================================
-- create_table(:feeds)
   -> 0.0025s
==  CreateFeeds: migrated (0.0026s) ===========================================

==  CreatePosts: migrating ====================================================
-- create_table(:posts)
   -> 0.0021s
==  CreatePosts: migrated (0.0023s) ===========================================

==  IndexPublishedInPosts: migrating ==========================================
-- add_index(:posts, [:published, :feed_id])
   -> 0.0015s
==  IndexPublishedInPosts: migrated (0.0017s) =================================

==  IndexFeedAuthor: migrating ================================================
-- add_index(:feeds, :author)
   -> 0.0007s
==  IndexFeedAuthor: migrated (0.0008s) =======================================

==  PublishedAsDatetime: migrating ============================================
-- change_column(:posts, :published, :datetime)
   -> 0.0086s
-- add_column(:posts, :updated, :datetime)
   -> 0.0006s
==  PublishedAsDatetime: migrated (0.0094s) ===================================
$ rake
...LOTS OF ERRORS...
activerecord-3.1.0/lib/active_record/base.rb:1083:
    in `method_missing': undefined method `named_scope' for #<Class:0x103875b28> (NoMethodError)

Now we’re getting some errors and are ready to get to work.

Update our code

Of course it doesn’t just work and we need to make a number of changes. The guides include an upgrade process section and the rails_upgrade plugin can help identify problems. Here are the ones I found.

Changing named_scope to scope

With Rails 3 the syntax for scopes changed from named_scope to scope. So we search-and-replace in our models. Once we do this the specs run, but many are failing. A lot are failing because the routes.rb syntax changed in Rails 3.

RAILS_ROOT => Rails.root

The constant RAILS_ROOT no longer exists so we need to search-and-replace that with Rails.root.

Routes.rb

# OLD config/routes.rb
ActionController::Routing::Routes.draw do |map|
  map.posts_by_author 'author/:id', :controller=>'posts', :action=>'by_author'
  map.posts_by_month 'month/:year/:month', :controller=>'posts', :action=>'by_month'
  map.atom_feed '/atom', :controller => "posts", :action=>'index', :format=>'atom'

  map.resources :feeds
  map.root :controller => "posts"
end

We change it to:

# NEW config/routes.rb
WayWeWork::Application.routes.draw do
  match 'author/:id' =>'posts#by_author', :as => :posts_by_author
  match 'month/:year/:month' =>'posts#by_month', :as => :posts_by_month
  match '/atom' => 'posts#index', :as => :atom_feed, :format => :atom

  resources :feeds

  root :to => 'posts#index'
end

RSpec Routing Specs

We have a bunch of routing specs that are failing because RSpec changed the syntax around routing specs.

# OLD spec/controllers/feeds_routing_spec.rb
it "should map #index" do
  route_for(:controller => "feeds", :action => "index").should == "/feeds"
end
it "should generate params for #index" do
  params_from(:get, "/feeds").should == {:controller => "feeds", :action => "index"}
end

The routing spec must be in folder called routing

# NEW spec/feeds_routing_spec.rb
it "should generate params for #index" do
  get("/feeds").should route_to('feeds#index')
end

RSpec stub!

Another syntax change in RSpec from using stubs to stub!

# OLD
it 'should ...' do
  #...
  feed.stubs(:puts)
  feed.get_latest
end

We get an error undefined method 'stubs' for \#<Feed:0x105d572e8>

# NEW
it 'should ...' do
  #...
  feed.stub!(:puts)
  feed.get_latest
end

View Specs

Hmm I’m surprised I had written view specs. Nowadays I would write cucumber features and never write view specs. That being said there were a few things I needed to change to get the view specs passing, all related to RSpec syntax changes.

I was passing @ variables to the view with assigns[:feed] = @feed = stub_model(Feed) and today you need to do that with assign(:feed, @feed = stub_model(Feed). Also when you call render "/feeds/edit.html.erb" it used to work but now tries to render a partial. Plain old render will work as long as you’ve put the filename in your describe like describe "/feeds/edit.html.erb" do

Deploying to Heroku

The old app was deployed to a VPS using Capistrano. Today I prefer to use Heroku and there were a few changes I had to make to get it working there.

  • Add pg to my Gemfile to support Postgresql
  • Configure assets (since we cannot write to the filesystem) ** Added config.assets.compile = true in my production.rb ** Added therubyracer to my Gemfile
  • Delete my old Capfile

Now I was able to deploy to heroku and the new app is up and running. All this done in less than a day!

Talk Ruby to a Ruby Class instead of JSON to an HTTP Service

5 months ago | Alex Rothenberg: Common Sense Software

Software as a service (SaaS) is a great thing. I love that other people are providing services and I don’t have to implement them myself. I can use Airbrake for error notifications and even Twitter for communication. It frees me to focus on what’s unique about my app. Its great that they all work with open standards like HTTP, JSON and XML. But what I like even more is not having to think about HTTP, JSON or XML! When writing a Ruby application I want to think about Ruby. The services I really like provide a gem that hides the transport api details and lets me write plain ruby.

If you are creating a service yourself or even using someone else’s service it’s not too hard to create your own client gem. Let’s look at an example how we can do that.

An Example: User Directory Service

Imagine you’re building a user directory that lets you list users, create users, update users, show users, delete users, you know, the standard RESTFUL actions. For instance we could create a user and list all users like this

[~]$ curl -F 'user[name]'='Mickey Mouse' http://user-directory.example.com/users.json
{"user":{"name":"Mickey Mouse","id":1001}}

[~]$ curl http://user-directory.example.com/users.json
[{"user":{"name":"Mickey Mouse","id":1001}}]

This is great to use from CURL but writing the code to talk http form fields and parse json will get pretty tiring. I’d much rather have a UserDirectory::User object that behaved similarly to an ActiveRecord model. When we’re done we want to be able to have it work this

user = UserDirectory::User.create(:name => 'Jenny', :phone => '867-5309')
# => #<UserDirectory::User:0x10239b0b8 @name="Jenny", @phone="867-5309", @id=1001>
user.name
# => 'Jenny'
user.id
# => 1001

UserDirectory::User.all
# => [ #<UserDirectory::User:0x102375638 @name="Jenny", @phone="867-5309", @id=1001>  ]

Building our client gem

We’ll want to wrap this client into a gem so it can be reused by many applications. I prefer to use bundler and its bundle gem command to start my gem. I’ll assume you know how to do that - check out this railscast if you’ve never done it before.

We’re going to get started by building a model class to represent the user without actually connecting it to our service yet.

module UserDirectory
  class User
    include ActiveModel::Validations
    validates_presence_of :name

    attr_accessor :name, :phone
    def initialize(attributes={})
      attributes.each do |name, value|
        send("#{name}=", value)
      end
    end
  end
end

I hope you didn’t think all model classes had to subclass ActiveRecord::Base! For me a model represents a concept in the application domain and does not have to map to a database table!

The magic in here is that we’re mixing in the ActiveModel::Validations module. This lets us add the same validations we would to a “regular” ActiveRecord model - in this case validates_presence_of :name. When Rails 3.0 came along much of the functionality of ActiveRecord was extracted into ActiveModel which lets you make any ruby object feel like ActiveRecord and as with most other topics Ryan Bates has put together a great railscast.

When we try it out it behaves as we expect.

# A valid user
user = UserDirectory::User.new(:name => 'alex')
# => #<UserDirectory::User:0x12b79fff0 @name="alex">
user.valid?
# => true

# an invalid user
user = UserDirectory::User.new(:phone => '555-1212')
# => #<UserDirectory::User:0x12b78ecf0 @phone="555-1212">
user.valid?
# => false
user.errors
# => #<ActiveModel::Errors:0x12b788f80 @base=#<UserDirectory::User:0x12b78ecf0
#      @errors=#<ActiveModel::Errors:0x12b788f80 ...>, @validation_context=nil, @phone="555-1212">,
#      @messages=#<OrderedHash {:name=>["can't be blank"]}>>

The next step is to make our model interact with the service using HTTP and JSON. I’m going to be using the HTTParty gem. It is much easier to use than net/http and automatically parses json or xml into a ruby hash for us.

First, we’ll add a create method that tells the service to create a new user as long as we pass our validations and then returns a User instance.

module UserDirectory
  class User
    # all the existing code and...

    def attributes
      { :name => name, :phone => phone }
    end

    include HTTParty
    base_uri 'http://user-directory.example.com'

    def self.create attributes
      user = new(attributes)
      return user unless user.valid?

      response = post('/users.json', :body => user.attributes)
      raise "#{response.code}: Better error handling please" unless response.success?
      self.new(response.parsed_response)
    end
  end
end

We can do the same for an all method that returns an array of all users stored in the service.

module UserDirectory
  class User
    # all the existing code and...

    def self.all
      response = get('/users.json')
      raise "#{response.code}: Better error handling please" unless response.success?
      response.parsed_response.map do |user_attributes|
        self.new user_attributes
      end
    end
  end
end

Let’s try it out.

# No users yet
UserDirectory::User.all
# => []

# Create a user
UserDirectory::User.create(:name => 'alex', :phone => '555-1212')
# => #<UserDirectory::User:0x12c52f0a8 @name="alex", @phone="555-1212">

# Now ask the service again
UserDirectory::User.all
# => [#<UserDirectory::User:0x12b8890d8 @name="alex", @phone="555-1212">]

This is great! We have a class that’s easy to use, behaves like a normal model but actually talks JSON and HTTP to a remote service. We could continue along the same lines to implement the other methods we need like find & destroy but I’m not going to bore you with that here. Instead I’ll switch gears and talk about building a fake service to eliminate the need to have the actual service running during development and testing.

Building a fake service

There are a few reasons we want to build a fake service.

  1. It’ll be faster to not make any network calls
  2. It’s a lot of overhead to start a local copy of the service during development
  3. It’ll be easier to test various failures (500 errors and the like)
  4. It allows us to setup and clear the data any way we want

How do we create a fake service? Luckily there’s a gem for that! The ShamRack gem intercepts http calls before they leave our app and redirects them to a local Rack App we’ll create. We’ll create a simple sinatra app that implements the UserDirectory Service API and embed it in our gem.

First things first, add the gem to our gem’s user_directory_client.gemspec

gem.add_dependency "sham_rack"

Now we can

require 'sinatra'
require 'sham_rack'

module UserDirectory
  class FakeService < Sinatra::Base

    ###################
    # ShamRack methods
    def self.activate!
      ShamRack.mount(self, "user-directory.example.com", 80)
    end
    def self.deactivate!
      ShamRack.unmount_all
    end


    ###################
    # Sinatra methods
    configure do
      set :raise_errors, true
      set :show_exceptions, false
    end

    USER_JSON = {:name=>'Jenny', :phone=>'867-5309'}.to_json

    get '/users.json' do
      content_type 'text/json'
      [USER_JSON] # :hardcoded list of 1 user
    end

    # :create new user
    post '/users.json' do
      content_type 'text/json'
      USER_JSON # pretend to create and return hardcoded user
    end
  end
end

Its very simple and will always return the same hardcoded user but that might be enough. We turn the fake service on and off with calls to UserDirectory::FakeService.activate! and UserDirectory::FakeService.deactivate!. Let’s take a look.

UserDirectory::FakeService.activate!
# => PeopleServices::FakePeopleService

UserDirectory::User.all
# => [#<UserDirectory::User:0x12c52f0a8 @name="jenny", @phone="867-5309">]

UserDirectory::User.create(:name => 'alex', :phone => '555-1212')
# => #<UserDirectory::User:0x12b8731e8 @name="jenny", @phone="867-5309">
UserDirectory::User.create(:name => 'pat')
# => #<UserDirectory::User:0x12b86cc58 @name="jenny", @phone="867-5309">

UserDirectory::User.all
# => [#<UserDirectory::User:0x12b865cf0 @name="jenny", @phone="867-5309">]

This is interesting. Its fast and eliminates the dependency, but its all hardcoded! When we created 2 users we still got the same “Jenny” user every time. The good news is the fake service is just a class we wrote so we can make is as complex as we need. Perhaps what we have here is enough for you and you’re all done but we’ll assume you want something a bit more realistic.

We’re going to create a quick array to simulate persisting our users.

require 'sinatra'
require 'sham_rack'

module UserDirectory
  class FakeService < Sinatra::Base

    # ShamRack methods ... remain unchanged

    # Sinatra methods ... changed to use our new "business logic"
    get '/users.json' do
      self.class.users.to_json
    end

    post '/users.json' do
      create_user(params)
    end

    ###################
    # Some "business logic"
    # the worlds simplest db :)
    def self.users
      @users ||= []
    end
    def create_user attributes
      attributes['id'] = rand(10000)
      self.class.users << attributes.dup
      attributes.to_json
    end
  end
end

One last time we’re going to try it out.

UserDirectory::FakeService.activate!
# => PeopleServices::FakePeopleService

# We start off empty
UserDirectory::User.all
# => []

# Create some users - the attributes correctly change
UserDirectory::User.create(:name => 'alex', :phone => '555-1212')
# => #<UserDirectory::User:0x12b85f300 @name="alex", @phone="555-1212">
UserDirectory::User.create(:name => 'pat')
# => #<UserDirectory::User:0x12b857880 @name="pat">

# Now we have our 2 users
UserDirectory::User.all
# => [#<UserDirectory::User:0x12b851318 @name="alex", @phone="555-1212">, #<UserDirectory::User:0x12b84adb0 @name="pat">]

This is now looking almost like a real service and will be very useful as we do our development. One last enhancement is that it’ll be nice to test what happens when the service has errors.

require 'sinatra'
require 'sham_rack'

module UserDirectory
  class FakeService < Sinatra::Base
    # Everything else unchanged...

    def self.fail_next_request!
      @fail_next_request = true
    end
    def self.should_fail_request?
      should_fail = @fail_next_request
      @fail_next_request = false
      should_fail
    end

    # Sinatra before filter
    before do
      halt 500, 'We were told to fail!' if self.class.should_fail_request?
    end
  end
end

One last time we’ll try it out.

UserDirectory::FakeService.activate!
# => true
UserDirectory::FakeService.fail_next_request!
# => true
 
UserDirectory::User.create :name => 'Alex'
RuntimeError: 500: We were told to fail!
	from /Users/alex/user_directory_client/lib/user_directory/user.rb:25:in `create'
	from (irb):56

Now go off and create a client for any service you create and be sure to include a fake service!

Talk Ruby to a Ruby Class instead of JSON to an HTTP Service

5 months ago | Alex Rothenberg: Common Sense Software

Software as a service (SaaS) is a great thing. I love that other people are providing services and I don’t have to implement them myself. I can use Airbrake for error notifications and even Twitter for communication. It frees me to focus on what’s unique about my app. Its great that they all work with open standards like HTTP, JSON and XML. But what I like even more is not having to think about HTTP, JSON or XML! When writing a Ruby application I want to think about Ruby. The services I really like provide a gem that hides the transport api details and lets me write plain ruby.

If you are creating a service yourself or even using someone else’s service it’s not too hard to create your own client gem. Let’s look at an example how we can do that.

An Example: User Directory Service

Imagine you’re building a user directory that lets you list users, create users, update users, show users, delete users, you know, the standard RESTFUL actions. For instance we could create a user and list all users like this

[~]$ curl -F 'user[name]'='Mickey Mouse' http://user-directory.example.com/users.json
{"user":{"name":"Mickey Mouse","id":1001}}

[~]$ curl http://user-directory.example.com/users.json
[{"user":{"name":"Mickey Mouse","id":1001}}]

This is great to use from CURL but writing the code to talk http form fields and parse json will get pretty tiring. I’d much rather have a UserDirectory::User object that behaved similarly to an ActiveRecord model. When we’re done we want to be able to have it work this

user = UserDirectory::User.create(:name => 'Jenny', :phone => '867-5309')
# => #<UserDirectory::User:0x10239b0b8 @name="Jenny", @phone="867-5309", @id=1001>
user.name
# => 'Jenny'
user.id
# => 1001

UserDirectory::User.all
# => [ #<UserDirectory::User:0x102375638 @name="Jenny", @phone="867-5309", @id=1001>  ]

Building our client gem

We’ll want to wrap this client into a gem so it can be reused by many applications. I prefer to use bundler and its bundle gem command to start my gem. I’ll assume you know how to do that - check out this railscast if you’ve never done it before.

We’re going to get started by building a model class to represent the user without actually connecting it to our service yet.

module UserDirectory
  class User
    include ActiveModel::Validations
    validates_presence_of :name

    attr_accessor :name, :phone
    def initialize(attributes={})
      attributes.each do |name, value|
        send("#{name}=", value)
      end
    end
  end
end

I hope you didn’t think all model classes had to subclass ActiveRecord::Base! For me a model represents a concept in the application domain and does not have to map to a database table!

The magic in here is that we’re mixing in the ActiveModel::Validations module. This lets us add the same validations we would to a “regular” ActiveRecord model - in this case validates_presence_of :name. When Rails 3.0 came along much of the functionality of ActiveRecord was extracted into ActiveModel which lets you make any ruby object feel like ActiveRecord and as with most other topics Ryan Bates has put together a great railscast.

When we try it out it behaves as we expect.

# A valid user
user = UserDirectory::User.new(:name => 'alex')
# => #<UserDirectory::User:0x12b79fff0 @name="alex">
user.valid?
# => true

# an invalid user
user = UserDirectory::User.new(:phone => '555-1212')
# => #<UserDirectory::User:0x12b78ecf0 @phone="555-1212">
user.valid?
# => false
user.errors
# => #<ActiveModel::Errors:0x12b788f80 @base=#<UserDirectory::User:0x12b78ecf0
#      @errors=#<ActiveModel::Errors:0x12b788f80 ...>, @validation_context=nil, @phone="555-1212">,
#      @messages=#<OrderedHash {:name=>["can't be blank"]}>>

The next step is to make our model interact with the service using HTTP and JSON. I’m going to be using the HTTParty gem. It is much easier to use than net/http and automatically parses json or xml into a ruby hash for us.

First, we’ll add a create method that tells the service to create a new user as long as we pass our validations and then returns a User instance.

module UserDirectory
  class User
    # all the existing code and...

    def attributes
      { :name => name, :phone => phone }
    end

    include HTTParty
    base_uri 'http://user-directory.example.com'

    def self.create attributes
      user = new(attributes)
      return user unless user.valid?

      response = post('/users.json', :body => user.attributes)
      raise "#{response.code}: Better error handling please" unless response.success?
      self.new(response.parsed_response)
    end
  end
end

We can do the same for an all method that returns an array of all users stored in the service.

module UserDirectory
  class User
    # all the existing code and...

    def self.all
      response = get('/users.json')
      raise "#{response.code}: Better error handling please" unless response.success?
      response.parsed_response.map do |user_attributes|
        self.new user_attributes
      end
    end
  end
end

Let’s try it out.

# No users yet
UserDirectory::User.all
# => []

# Create a user
UserDirectory::User.create(:name => 'alex', :phone => '555-1212')
# => #<UserDirectory::User:0x12c52f0a8 @name="alex", @phone="555-1212">

# Now ask the service again
UserDirectory::User.all
# => [#<UserDirectory::User:0x12b8890d8 @name="alex", @phone="555-1212">]

This is great! We have a class that’s easy to use, behaves like a normal model but actually talks JSON and HTTP to a remote service. We could continue along the same lines to implement the other methods we need like find & destroy but I’m not going to bore you with that here. Instead I’ll switch gears and talk about building a fake service to eliminate the need to have the actual service running during development and testing.

Building a fake service

There are a few reasons we want to build a fake service.

  1. It’ll be faster to not make any network calls
  2. It’s a lot of overhead to start a local copy of the service during development
  3. It’ll be easier to test various failures (500 errors and the like)
  4. It allows us to setup and clear the data any way we want

How do we create a fake service? Luckily there’s a gem for that! The ShamRack gem intercepts http calls before they leave our app and redirects them to a local Rack App we’ll create. We’ll create a simple sinatra app that implements the UserDirectory Service API and embed it in our gem.

First things first, add the gem to our gem’s user_directory_client.gemspec

gem.add_dependency "sham_rack"

Now we can

require 'sinatra'
require 'sham_rack'

module UserDirectory
  class FakeService < Sinatra::Base

    ###################
    # ShamRack methods
    def self.activate!
      ShamRack.mount(self, "user-directory.example.com", 80)
    end
    def self.deactivate!
      ShamRack.unmount_all
    end


    ###################
    # Sinatra methods
    configure do
      set :raise_errors, true
      set :show_exceptions, false
    end

    USER_JSON = {:name=>'Jenny', :phone=>'867-5309'}.to_json

    get '/users.json' do
      content_type 'text/json'
      [USER_JSON] # :hardcoded list of 1 user
    end

    # :create new user
    post '/users.json' do
      content_type 'text/json'
      USER_JSON # pretend to create and return hardcoded user
    end
  end
end

Its very simple and will always return the same hardcoded user but that might be enough. We turn the fake service on and off with calls to UserDirectory::FakeService.activate! and UserDirectory::FakeService.deactivate!. Let’s take a look.

UserDirectory::FakeService.activate!
# => PeopleServices::FakePeopleService

UserDirectory::User.all
# => [#<UserDirectory::User:0x12c52f0a8 @name="jenny", @phone="867-5309">]

UserDirectory::User.create(:name => 'alex', :phone => '555-1212')
# => #<UserDirectory::User:0x12b8731e8 @name="jenny", @phone="867-5309">
UserDirectory::User.create(:name => 'pat')
# => #<UserDirectory::User:0x12b86cc58 @name="jenny", @phone="867-5309">

UserDirectory::User.all
# => [#<UserDirectory::User:0x12b865cf0 @name="jenny", @phone="867-5309">]

This is interesting. Its fast and eliminates the dependency, but its all hardcoded! When we created 2 users we still got the same “Jenny” user every time. The good news is the fake service is just a class we wrote so we can make is as complex as we need. Perhaps what we have here is enough for you and you’re all done but we’ll assume you want something a bit more realistic.

We’re going to create a quick array to simulate persisting our users.

require 'sinatra'
require 'sham_rack'

module UserDirectory
  class FakeService < Sinatra::Base

    # ShamRack methods ... remain unchanged

    # Sinatra methods ... changed to use our new "business logic"
    get '/users.json' do
      self.class.users.to_json
    end

    post '/users.json' do
      create_user(params)
    end

    ###################
    # Some "business logic"
    # the worlds simplest db :)
    def self.users
      @users ||= []
    end
    def create_user attributes
      attributes['id'] = rand(10000)
      self.class.users << attributes.dup
      attributes.to_json
    end
  end
end

One last time we’re going to try it out.

UserDirectory::FakeService.activate!
# => PeopleServices::FakePeopleService

# We start off empty
UserDirectory::User.all
# => []

# Create some users - the attributes correctly change
UserDirectory::User.create(:name => 'alex', :phone => '555-1212')
# => #<UserDirectory::User:0x12b85f300 @name="alex", @phone="555-1212">
UserDirectory::User.create(:name => 'pat')
# => #<UserDirectory::User:0x12b857880 @name="pat">

# Now we have our 2 users
UserDirectory::User.all
# => [#<UserDirectory::User:0x12b851318 @name="alex", @phone="555-1212">, #<UserDirectory::User:0x12b84adb0 @name="pat">]

This is now looking almost like a real service and will be very useful as we do our development. One last enhancement is that it’ll be nice to test what happens when the service has errors.

require 'sinatra'
require 'sham_rack'

module UserDirectory
  class FakeService < Sinatra::Base
    # Everything else unchanged...

    def self.fail_next_request!
      @fail_next_request = true
    end
    def self.should_fail_request?
      should_fail = @fail_next_request
      @fail_next_request = false
      should_fail
    end

    # Sinatra before filter
    before do
      halt 500, 'We were told to fail!' if self.class.should_fail_request?
    end
  end
end

One last time we’ll try it out.

UserDirectory::FakeService.activate!
# => true
UserDirectory::FakeService.fail_next_request!
# => true
 
UserDirectory::User.create :name => 'Alex'
RuntimeError: 500: We were told to fail!
	from /Users/alex/user_directory_client/lib/user_directory/user.rb:25:in `create'
	from (irb):56

Now go off and create a client for any service you create and be sure to include a fake service!

Running a Private GemServer inside the Firewall

5 months ago | Alex Rothenberg: Common Sense Software

rubygems.org has made it so easy to publish a gem for the world to use but what do you do when your gem is proprietary and you only want to publish it within your company?

This is something I’ve just been through at my company and thought I’d share the steps I went through. We need to

  1. Setup an inside-the-firewall gem server
  2. Configure our gems to deploy to it
  3. Configure our apps to use it

Setup an inside-the-firewall gem server

The first thing you have to decide is what gemserver to use. Rubygems.org has a helpful page called running your own gemserver that basically lists 3 choices in a goldilocks situation.

  1. too small - gem server is a command built into rubygems

    This works but you need to log onto the server to install a new gem and it serves all gems on the system not just your proprietary ones

  2. too big - rubygems.org is open source so we could deploy it on our own server

    This seems pretty complex to setup and even they tell you to “consider checking out Geminabox”

  3. just right - gem in a box is a simple sinatra app to allow you to host your own in-house gems

    This is easy to setup, has a web interface and supports a command line to remotely publish new gems.

geminabox is what I decided to go with.

The readme on github describes the server setup for geminabox and it just worked. The only thing to keep in mind is that you cannot use bundler as then you will only serve the gems in the bundle instead of the gems you publish. I spent some time adding bundler before realizing that was a bad idea and backing it out.

Once geminabox is up and running you can view your gems at your internal url and you’ll see the gem server homepage showing you it has no gems.

New Gem Server

The easiest thing is to add a new gem by clicking “Upload Another Gem” and selecting a .gem file from your hard drive (I picked diagnostics-0.0.1.gem in the image below).

Upload A Gem Manuallly

Once you click uppload you should see your gem on the page.

Gem Server With A Gem

At this point we could start using this gem server in our apps but before we talk about that let’s automate the manual process we just went through to add a gem.

Configure our gems to deploy to the gem server

I’ve been using bundler to create my gems with the bundle gem command and one of the features that gives you is a set of nice rake tasks. Check out the New Gem with Bundler Railscast to learn how it works.

$ bundle gem my_awesome_gem
      create  my_awesome_gem/Gemfile
      create  my_awesome_gem/Rakefile
      create  my_awesome_gem/.gitignore
      create  my_awesome_gem/my_awesome_gem.gemspec
      create  my_awesome_gem/lib/my_awesome_gem.rb
      create  my_awesome_gem/lib/my_awesome_gem/version.rb
Initializating git repo in /Users/alex/my_awesome_gem

Let’s look at the tasks we’ve got.

$ cd my_awesome_gem
$ rake -T
rake build    # Build my_awesome_gem-0.0.1.gem into the pkg directory
rake install  # Build and install my_awesome_gem-0.0.1.gem into system gems
rake release  # Create tag v0.0.1 and build and push my_awesome_gem-0.0.1.gem to Rubygems

rake build and rake install do their work locally but rake release is what you call when you’re done and ready to release your gem into the wild. This task will push your changes to github, create a git tag, build your gem package and deploy it to http://rubygems.org. We need to do something to change that last part so it deploys to our private gem server instead of rubygems.org.

Let’s spend some time looking into bundler to figure out how rake release works. The magic all happens inside a file lib/bundler/gem_helper.rb

  • It defines a :release rake task
  • Which calls release_gem
  • Which calls rubygem_push
  • Finally this will call gem push pkg/my_awesome_gem-0.0.1.gem which pushes to http://rubygems.org. We’ve found the behavior we need to change.

geminabox adds a custom rubygems command called inabox so you can deploy a gem with the command gem inabox pkg/my-awesome-gem-1.0.gem. Unfortunately bundler does not seem to have a convenient way to change this so we’re going to monkey patch bundler Bundler::GemHelper#rugygem_push method to use the geminabox command instead. (please let me know if you have a better idea)

We’ll add our monkey patch to our Rakefile since its called by a rake command.

# Rakefile in your my_awesome_gem gem

# Monkey patch Bundler gem_helper so we release to our gem server instead of rubygems.org
module Bundler
  class GemHelper
    def rubygem_push(path)
      gem_server_url = 'http://gems.intranet.mycompany.com'
      sh("gem inabox '#{path}' --host #{gem_server_url}")
      Bundler.ui.confirm "Pushed #{name} #{version} to #{gem_server_url}"
    end
  end
end

You can see this will call gem inabox ... so we also need to add geminabox to our gem’s bundle. We do this in the .gemspec as a development dependency

# my_awesome_gem.gemspec in your gem

Gem::Specification.new do |s|
  .. lots of other stuff ...

  s.add_development_dependency "geminabox"
end

Now when we call rake release it will push the gem to our private server instead of the public one. Let’s see:

$ rake release
my_awesome_gem 0.0.1 built to pkg/my_awesome_gem-0.0.1.gem
Tagged v0.0.1
Pushed git commits and tags
Pushed my_awesome_gem 0.0.1 to http://gems.intranet.mycompany.com

Now when we go to the gem server site, we can see our new awesome gem in the list

Gem Server With Awesome Gem

The gem is there an you can use install it with a command like gem install my_awesome_gem --source http://gems.intranet.mycompany.com

Using your Gem Server from an application

We’ve just seen how we can use the source option to tell rubygems where to look when installing our gem by hand, but in a modern application we all use bundler and a Gemfile to manage our gems so how do we tell bundler to user our private gemserver for our private gems? Its super simple, you just need to add a source to the top of your Gemfile

source "http://gems.intranet.mycompany.com/"
source :rubygems

# regular old gems come from rubygems.org
gem "rails"
gem "rack"
gem "haml"

# my private gem comes from my private gemserver
gem 'diagnostics'

Now when we run bundle it looks in our private gem server as well as the public rubygems.org. Now that that you’ve got my_awesoem_gem you’re ready to add awesomeness to your app.

$ bundle
Fetching source index for http://gems.intranet.mycompany.com/
Fetching source index for http://rubygems.org/
Using rake (0.9.2)
Using activesupport (3.1.0)
Installing my_awesome_gem (0.0.1)
...etc..

Running a Private GemServer inside the Firewall

5 months ago | Alex Rothenberg: Common Sense Software

rubygems.org has made it so easy to publish a gem for the world to use but what do you do when your gem is proprietary and you only want to publish it within your company?

This is something I’ve just been through at my company and thought I’d share the steps I went through. We need to

  1. Setup an inside-the-firewall gem server
  2. Configure our gems to deploy to it
  3. Configure our apps to use it

Setup an inside-the-firewall gem server

The first thing you have to decide is what gemserver to use. Rubygems.org has a helpful page called running your own gemserver that basically lists 3 choices in a goldilocks situation.

  1. too small - gem server is a command built into rubygems

    This works but you need to log onto the server to install a new gem and it serves all gems on the system not just your proprietary ones

  2. too big - rubygems.org is open source so we could deploy it on our own server

    This seems pretty complex to setup and even they tell you to “consider checking out Geminabox”

  3. just right - gem in a box is a simple sinatra app to allow you to host your own in-house gems

    This is easy to setup, has a web interface and supports a command line to remotely publish new gems.

geminabox is what I decided to go with.

The readme on github describes the server setup for geminabox and it just worked. The only thing to keep in mind is that you cannot use bundler as then you will only serve the gems in the bundle instead of the gems you publish. I spent some time adding bundler before realizing that was a bad idea and backing it out.

Once geminabox is up and running you can view your gems at your internal url and you’ll see the gem server homepage showing you it has no gems.

New Gem Server

The easiest thing is to add a new gem by clicking “Upload Another Gem” and selecting a .gem file from your hard drive (I picked diagnostics-0.0.1.gem in the image below).

Upload A Gem Manuallly

Once you click uppload you should see your gem on the page.

Gem Server With A Gem

At this point we could start using this gem server in our apps but before we talk about that let’s automate the manual process we just went through to add a gem.

Configure our gems to deploy to the gem server

I’ve been using bundler to create my gems with the bundle gem command and one of the features that gives you is a set of nice rake tasks. Check out the New Gem with Bundler Railscast to learn how it works.

$ bundle gem my_awesome_gem
      create  my_awesome_gem/Gemfile
      create  my_awesome_gem/Rakefile
      create  my_awesome_gem/.gitignore
      create  my_awesome_gem/my_awesome_gem.gemspec
      create  my_awesome_gem/lib/my_awesome_gem.rb
      create  my_awesome_gem/lib/my_awesome_gem/version.rb
Initializating git repo in /Users/alex/my_awesome_gem

Let’s look at the tasks we’ve got.

$ cd my_awesome_gem
$ rake -T
rake build    # Build my_awesome_gem-0.0.1.gem into the pkg directory
rake install  # Build and install my_awesome_gem-0.0.1.gem into system gems
rake release  # Create tag v0.0.1 and build and push my_awesome_gem-0.0.1.gem to Rubygems

rake build and rake install do their work locally but rake release is what you call when you’re done and ready to release your gem into the wild. This task will push your changes to github, create a git tag, build your gem package and deploy it to http://rubygems.org. We need to do something to change that last part so it deploys to our private gem server instead of rubygems.org.

Let’s spend some time looking into bundler to figure out how rake release works. The magic all happens inside a file lib/bundler/gem_helper.rb

  • It defines a :release rake task
  • Which calls release_gem
  • Which calls rubygem_push
  • Finally this will call gem push pkg/my_awesome_gem-0.0.1.gem which pushes to http://rubygems.org. We’ve found the behavior we need to change.

geminabox adds a custom rubygems command called inabox so you can deploy a gem with the command gem inabox pkg/my-awesome-gem-1.0.gem. Unfortunately bundler does not seem to have a convenient way to change this so we’re going to monkey patch bundler Bundler::GemHelper#rugygem_push method to use the geminabox command instead. (please let me know if you have a better idea)

We’ll add our monkey patch to our Rakefile since its called by a rake command.

# Rakefile in your my_awesome_gem gem

# Monkey patch Bundler gem_helper so we release to our gem server instead of rubygems.org
module Bundler
  class GemHelper
    def rubygem_push(path)
      gem_server_url = 'http://gems.intranet.mycompany.com'
      sh("gem inabox '#{path}' --host #{gem_server_url}")
      Bundler.ui.confirm "Pushed #{name} #{version} to #{gem_server_url}"
    end
  end
end

You can see this will call gem inabox ... so we also need to add geminabox to our gem’s bundle. We do this in the .gemspec as a development dependency

# my_awesome_gem.gemspec in your gem

Gem::Specification.new do |s|
  .. lots of other stuff ...

  s.add_development_dependency "geminabox"
end

Now when we call rake release it will push the gem to our private server instead of the public one. Let’s see:

$ rake release
my_awesome_gem 0.0.1 built to pkg/my_awesome_gem-0.0.1.gem
Tagged v0.0.1
Pushed git commits and tags
Pushed my_awesome_gem 0.0.1 to http://gems.intranet.mycompany.com

Now when we go to the gem server site, we can see our new awesome gem in the list

Gem Server With Awesome Gem

The gem is there an you can use install it with a command like gem install my_awesome_gem --source http://gems.intranet.mycompany.com

Using your Gem Server from an application

We’ve just seen how we can use the source option to tell rubygems where to look when installing our gem by hand, but in a modern application we all use bundler and a Gemfile to manage our gems so how do we tell bundler to user our private gemserver for our private gems? Its super simple, you just need to add a source to the top of your Gemfile

source "http://gems.intranet.mycompany.com/"
source :rubygems

# regular old gems come from rubygems.org
gem "rails"
gem "rack"
gem "haml"

# my private gem comes from my private gemserver
gem 'diagnostics'

Now when we run bundle it looks in our private gem server as well as the public rubygems.org. Now that that you’ve got my_awesoem_gem you’re ready to add awesomeness to your app.

$ bundle
Fetching source index for http://gems.intranet.mycompany.com/
Fetching source index for http://rubygems.org/
Using rake (0.9.2)
Using activesupport (3.1.0)
Installing my_awesome_gem (0.0.1)
...etc..

How Bundler Groups relate to the Rails Environment

7 months ago | Alex Rothenberg: Common Sense Software

Recently I’ve seen more and more Gemfiles that organize gems into groups and it got me wondering how bundler knows which groups to load. For the most part two things happen

  1. At install time - Bundler includes a capistrano task that installs all gems except those only in the development or test groups on your server
  2. At execution time - Rails tells bundler to load the default gems and those specific to your environment (development, staging or production)

How Bundler installs gems into your bundle

To tell bundler to use bundler on the server all you need to do is add the one line below to your Capfile

require 'bundler/capistrano'

This creates a capistrano task bundle:install that ultimately runs something like the command below on your server

bundle install --gemfile /srv/my_app/releases/20110715204318/Gemfile --path /srv/my_app/shared/bundle
               --deployment --quiet --without development test

Okay so it ran a bundle install but what really happened? Let’s take that command one piece at a time.

  • --gemfile /srv/my_app/releases/20110715204318/Gemfile tells it to use our Gemfile, that makes sense.

  • --path /srv/my_app/shared/bundle tells it where to put the bundle. Let’s see what that means.

    It looks like it created all the rubygems directories for to isolate the gems for this project (very similarly to rvm gemsets)

    $ ls /srv/my_app/shared/bundle/
    ruby
    $ ls /srv/my_app/shared/bundle/ruby/
    1.8
    $ ls /srv/my_app/shared/bundle/ruby/1.8/
    bin  cache  doc  gems  specifications
    
  • --quiet hmm what else can I say

  • --without development test Aha so here’s where it tells bundler to skip the development and test groups. so allall gems outside a group or in a group other than development or test are installed.

How does Bundler remember these settings when it loads Rails and tries to load the bundle? It saves them away in a .bundle directory cat .bundle/config shows us

---
BUNDLE_FROZEN: "1"
BUNDLE_DISABLE_SHARED_GEMS: "1"
BUNDLE_WITHOUT: development:test
BUNDLE_PATH: /srv/my_app/shared/bundle

Now we understand how Bundler and Capistrano work together during a deployment to setup the bundle and install gems on the server. Let’s take a look at what happens when our app starts up.

How Rails and Bundler load your gems according to the Rails Environment

In your config/application.rb, right near the top, you have a line like this.

# If you have a Gemfile, require the gems listed there, including any gems
# you've limited to :test, :development, or :production.
Bundler.require(:default, Rails.env) if defined?(Bundler)

Rails tells bundler to require all the gems in the :default group and also the current Rails.env group. It uses the .bundle/config file to know where the gems are installed and find them. So that’s how the gems appropriate for your environment get automatically loaded when Rails starts.

What if you create a gem group that doesn’t correspond to any Rails env?

This is the problem that started me down this investigation. I came across a Gemfile with a group called cruise like this

group :cruise do
  gem 'metric_fu'
end

It was working meaning our cruise server ran metric_fu but why?

  1. We weren’t using capistrano to run bundle install and instead just checked whether we were on our cruise server and ran the command bundle install in our Rakefile. Aside: We are looking into Jenkins as a continuous integration server that supports bundler This explains why the metric_fu was installed into our bundle (there was no --without so all gems are installed)

  2. When our Rails app starts it would not load metric_fu becuase Rails.env will never be cruise when the application.rb line Bundler.require(:default, Rails.env) runs. We had worked around that by doing the require ourselves.

   require 'metric_fu'
   

While this does work in that our cruise build works it has the downside of installing metric_fu (and all the gems it depends on) on our production server! That’s because the bundler/capistrano task installs all gems not marked development or test and since metric_fu is marked cruise it gets installed. Now Rails will not load it so its not that bad but its still not good. We can take a quick look on our server to verify

$ ls /srv/my_app/shared/bundle/ruby/1.8/specifications/metric_fu-2.0.1.gemspec
shared/bundle/ruby/1.8/specifications/metric_fu-2.0.1.gemspec
$ ls /srv/my_app/shared/bundle/ruby/1.8/gems/metric_fu-2.0.1
HISTORY  lib  MIT-LICENSE  Rakefile  README  spec  tasks  TODO

Fortunately this is really simple to fix, we just need to change our Gemfile and move metric_fu into the test group

group :test do
  gem 'metric_fu'
end

My advice it do not create any gem groups that do not correspond to your Rails environments as that seems to be what the bundler-capistrano and bundler-rails integrations expect.

How Bundler Groups relate to the Rails Environment

7 months ago | Alex Rothenberg: Common Sense Software

Recently I’ve seen more and more Gemfiles that organize gems into groups and it got me wondering how bundler knows which groups to load. For the most part two things happen

  1. At install time - Bundler includes a capistrano task that installs all gems except those only in the development or test groups on your server
  2. At execution time - Rails tells bundler to load the default gems and those specific to your environment (development, staging or production)

How Bundler installs gems into your bundle

To tell bundler to use bundler on the server all you need to do is add the one line below to your Capfile

require 'bundler/capistrano'

This creates a capistrano task bundle:install that ultimately runs something like the command below on your server

bundle install --gemfile /srv/my_app/releases/20110715204318/Gemfile --path /srv/my_app/shared/bundle
               --deployment --quiet --without development test

Okay so it ran a bundle install but what really happened? Let’s take that command one piece at a time.

  • --gemfile /srv/my_app/releases/20110715204318/Gemfile tells it to use our Gemfile, that makes sense.

  • --path /srv/my_app/shared/bundle tells it where to put the bundle. Let’s see what that means.

    It looks like it created all the rubygems directories for to isolate the gems for this project (very similarly to rvm gemsets)

    $ ls /srv/my_app/shared/bundle/
    ruby
    $ ls /srv/my_app/shared/bundle/ruby/
    1.8
    $ ls /srv/my_app/shared/bundle/ruby/1.8/
    bin  cache  doc  gems  specifications
    
  • --quiet hmm what else can I say

  • --without development test Aha so here’s where it tells bundler to skip the development and test groups. so allall gems outside a group or in a group other than development or test are installed.

How does Bundler remember these settings when it loads Rails and tries to load the bundle? It saves them away in a .bundle directory cat .bundle/config shows us

---
BUNDLE_FROZEN: "1"
BUNDLE_DISABLE_SHARED_GEMS: "1"
BUNDLE_WITHOUT: development:test
BUNDLE_PATH: /srv/my_app/shared/bundle

Now we understand how Bundler and Capistrano work together during a deployment to setup the bundle and install gems on the server. Let’s take a look at what happens when our app starts up.

How Rails and Bundler load your gems according to the Rails Environment

In your config/application.rb, right near the top, you have a line like this.

# If you have a Gemfile, require the gems listed there, including any gems
# you've limited to :test, :development, or :production.
Bundler.require(:default, Rails.env) if defined?(Bundler)

Rails tells bundler to require all the gems in the :default group and also the current Rails.env group. It uses the .bundle/config file to know where the gems are installed and find them. So that’s how the gems appropriate for your environment get automatically loaded when Rails starts.

What if you create a gem group that doesn’t correspond to any Rails env?

This is the problem that started me down this investigation. I came across a Gemfile with a group called cruise like this

group :cruise do
  gem 'metric_fu'
end

It was working meaning our cruise server ran metric_fu but why?

  1. We weren’t using capistrano to run bundle install and instead just checked whether we were on our cruise server and ran the command bundle install in our Rakefile. Aside: We are looking into Jenkins as a continuous integration server that supports bundler This explains why the metric_fu was installed into our bundle (there was no --without so all gems are installed)

  2. When our Rails app starts it would not load metric_fu becuase Rails.env will never be cruise when the application.rb line Bundler.require(:default, Rails.env) runs. We had worked around that by doing the require ourselves.

   require 'metric_fu'
   

While this does work in that our cruise build works it has the downside of installing metric_fu (and all the gems it depends on) on our production server! That’s because the bundler/capistrano task installs all gems not marked development or test and since metric_fu is marked cruise it gets installed. Now Rails will not load it so its not that bad but its still not good. We can take a quick look on our server to verify

$ ls /srv/my_app/shared/bundle/ruby/1.8/specifications/metric_fu-2.0.1.gemspec
shared/bundle/ruby/1.8/specifications/metric_fu-2.0.1.gemspec
$ ls /srv/my_app/shared/bundle/ruby/1.8/gems/metric_fu-2.0.1
HISTORY  lib  MIT-LICENSE  Rakefile  README  spec  tasks  TODO

Fortunately this is really simple to fix, we just need to change our Gemfile and move metric_fu into the test group

group :test do
  gem 'metric_fu'
end

My advice it do not create any gem groups that do not correspond to your Rails environments as that seems to be what the bundler-capistrano and bundler-rails integrations expect.

Backbone.js Makes Building JavaScript Applications Fun

about 1 year ago | Alex Rothenberg: Common Sense Software

Like many developers I’ve had a long, complicated relationship with Javascript. Especially with libraries like jquery it’s incredibly easy to add interesting behavior to your pages, but unless you’re very careful its also likely that you’ll end up with a mess of spaghetti javascript. I know as I’ve gotten myself into that mess and abandoned many projects because they were just too hard to change. All this has changed with some of new libraries out there that help you write your javascript following the MVC pattern.

Today I’m going to talk about backbone.js and show how it helped me and two friends build a rich one-page application to understand an exception stack trace. Backbone.js describes itself as supplying structure to JavaScript-heavy applications by providing models with key-value binding and custom events, collections with a rich API of enumerable functions, views with declarative event handling, and connects it all to your existing application over a RESTful JSON interface.

There are a number of tutorials and examples available out there to get you started. The basic idea is to uses backbone.js to organize your code into models, views and controllers.

What is CodeBuddy

CodeBuddy is the application I’ll describe. It helps you navigate an exception stack raised by your Rails app or any other stack you paste in. I worked on it with Pat Shaughnessy and Daniel Higginbotham and Pat produced a super article describing what CodeBuddy is and how to use it.

There is some server-side code that replaces the Rails Show Exceptions page and syntax highlights a snippet of source code for each line in the stack but for this article I’m going to ignore that and focus on the interactive page and the javascript behind it.

Below is a picture showing a stack trace the background and the code snippet for the currently selected line in front. Now a picture is okay but to really get a sense of it I suggest you follow this link to experience Code Buddy in action - try pressing ↑ or ↓ or double clicking a few lines in the stack then press s.

Code Buddy

There’s a lot going on here and if we had tried building this before discovering backbone.js we probably would have had created a mess - mixing javascript and html that quickly would have become hard to change. I’m going to show you how backbone let us separate the models from the views and create something that was not hard to grow.

Let’s get into the technical details!

Organizing our data in Backbone Models

We want to follow good OO design principles and thinking about this our exception stack is really just a few objects:

  • A Stack

    • has many Addresses
    • knows which Address is selected
  • Each Address has

    • path to a file
    • line number
    • snippet of code

We can build this as a JSON object like

var stackJson = {
  "stack_frames": [
    { "path": "/Users/alex/ruby/github/test_app/app/controllers/users_controller.rb",
      "line": 5,
      "code": "class UsersController < ApplicationController\n..."
    },
    { "path": "/Users/alex/.rvm/.../lib/action_controller/metal/implicit_render.rb",
      "line": 4,
      "code": "module ActionController\n..."
    }
  ],
  "selected": 0
}

The first step is to turn this into backbone models.

// Stack, Address and Addresses
CodeBuddy.backbone.Stack = Backbone.Model.extend({
  initialize: function() {
    this.set({
      addresses: new CodeBuddy.backbone.Addresses(this.get('stack_frames')) 
    })
  }
})

CodeBuddy.backbone.Address = Backbone.Model.extend({
})

CodeBuddy.backbone.Addresses = Backbone.Collection.extend({
  model:CodeBuddy.backbone.Address
})

The way we build a model in backbone is by extending Backbone.Model. When extending we can add custom behavior if we want. In our example we tell the Stack to contain a collection of Address objects in the Addresses collection. We use the backbone collection framework to define Addresses and tell it that it is a collection of Address model objects. These models will use the default backbone behavior for the rest which includes read/write access to its properties get or set.

Now that they’re defined, we’re ready to interact with these models. For example below is what you’d see using the Chrome javascript console (I’m showing the output as comments for readability)

// Using our models in a console
CodeBuddy.stack = new CodeBuddy.backbone.Stack({
  "stack_frames": [
    { "path": "/Users/alex/ruby/github/test_app/app/controllers/users_controller.rb",
      "line": 5,
      "code": "class UsersController < ApplicationController\n..."
    },
    { "path": "/Users/alex/.rvm/.../lib/action_controller/metal/implicit_render.rb",
      "line": 4,
      "code": "module ActionController\n..."
    }
  ],
  "selected": 0
})
// inherits.child

CodeBuddy.stack.get('addresses')
// inherits.child

CodeBuddy.stack.get('addresses').first().get('path')
// "/Users/alex/ruby/github/test_app/app/controllers/users_controller.rb"

CodeBuddy.stack.get('addresses').first().get('code')
// "class UsersController < ApplicationController
// ..."

We now have an object hierarchy with default behavior which we can (and will) extend in a bit but first let’s build some views and get a page we can look at.

Building the UI with Backbone Views

We’re going to build

  • A StackView that contains many AddressViews
  • Each AddressView will display a single line from the stack.
  • Each view will be tied to one of our model objects

It will look something like this:

Code Buddy views on the page

We can start with a StackView that’s tied to the Stack model. Creating a backbone view is very similar to how we created our models - we extend Backbone.View and can override behavior if we want.

CodeBuddy.backbone.StackView = Backbone.View.extend({

  el: $("#stack"),

  initialize: function() {
    this.model.get('addresses').each(this.addOneAddress);
  },
  
  addOneAddress: function(address, index) {
    var view = new CodeBuddy.backbone.AddressView({model: address});
    this.$("#stack").append(view.render().el);
  }
})

In this case we overrode the initializer function to create an AddressView for each address. It also uses jQuery to add the AddressView's html within the page’s #stack element.

To do the iteration we use another powerful javascript library called underscore.js. Underscore.js gives us a ruby-like collection methods letting us write .each(this.addOneAddress). This will iterate over all the addressses calling the addOneAddress function on each one. Underscore.js also gives us erb-like templating we’ll use in the AddressView…let’s take a look at that view.

CodeBuddy.backbone.AddressView = Backbone.View.extend({
  tagName:  "li",

  template: _.template("<span class='container'><%= path %>:<%= line%></span>"),

  initialize: function() {
  },

  render: function() {
    var html = this.template(this.model.toJSON())
    $(this.el).html(html);
    return this;
  }
})

You can see the template and how it does look like erb. It defines the html that will be displayed for each address and its able to access properties in the model like path and line. The template gets applied in the render function with the line this.template(this.model.toJSON()). Finally, tagName is used to wrap this html in an li tag.

Now we can put it all together to see on a page. We can start with a toplevel page that has an element with stack, includes our models and views and tells them to load.

<html>
<body>
  <ul id="stack"></ul>
  <script src="javascripts/code_buddy.js" type="text/javascript"></script>
  <script>
      CodeBuddy.setup({
        "stack_frames": [
          { "path": "/Users/alex/ruby/github/test_app/app/controllers/users_controller.rb",
            "line": 5,
            "code": "class UsersController < ApplicationController\n..."
          },
          { "path": "/Users/alex/.rvm/.../lib/action_controller/metal/implicit_render.rb",
            "line": 4,
            "code": "module ActionController\n..."
          }
        ],
        "selected": 0
      })
  </script>
</body>
</html>

After the page loads and the views render it changes the #stack div to have all this

<ul id="stack">
  <li>
    <span class="container">/Users/alex/ruby/github/test_app/app/controllers/users_controller.rb:5</span>
  </li>
  <li>
    <span class="container">/Users/alex/.rvm/.../lib/action_controller/metal/implicit_render.rb:4</span>
  </li>
</ul>

So far this is a lot of framework and structure for a simple page but now is when it gets interesting and backbone reveals its true power!

Showing code for the selected address

Let’s say we want to mark one address selected and show the code for that one.

Showing Code for selected Addresss

We can start by adding an element to the html called code-viewer

<html>
<body>
  <ul id="stack"></ul>
  <div id="code-viewer">
    <div id="code"></div>
  </div>
  <!-- all the javascript omitted for clarity -->
</body>
</html>

Now we want to create a StackView to put the right code in there

CodeBuddy.backbone.CodeView = Backbone.View.extend({
  el:$("#code-viewer"),
  
  initialize: function() {
  },
  
  render: function() {
    this.$("#code").html(CodeBuddy.stack.selectedAddress().get('code'))
  }
})

In this view we used our models to find which address was selected CodeBuddy.stack.selectedAddress() so let’s add that method.

CodeBuddy.backbone.Stack = Backbone.Model.extend({
  initialize: function() {
    // same code as before
  },

  addresses: function() {
    return this.get('addresses')
  },

  selectedAddress: function() {
    var selected = this.get('selected')
    return this.addresses().at(selected)
  }
})

That’s it, now we have a code view on top of the stack view and our code is still clean and well organized. I’m not afraid to keep going and make our next change.

Changing the selected address

Oh, we just got a new requirement to be able to scroll up and down through the stack and see the code window change as we go.

We add some more functions to our Stack Model so we can change the selection and we also tell it to call the selectionChanged function whenever the selected property changes. Calling a function on a property change is something that’s given to us with backbone events. Getting the view to update itself is also easy we just tell it to with CodeBuddy.codeView.render()

CodeBuddy.backbone.Stack = Backbone.Model.extend({
  initialize: function() {
    this.bind('change:selected', this.selectionChanged);
    // same code as before
  },

  setSelection: function(newSelected) {
    if (newSelected >= 0 && newSelected < this.addresses().size()) {
      this.set({ selected: newSelected })
    }
  },
  
  selectPrevious: function() {
    this.setSelection(this.get('selected') - 1)
  },
  
  selectNext: function() {
    this.setSelection(this.get('selected') + 1)
  }, 

  selectionChanged: function(x) {
    this.addresses().at(x.previousAttributes().selected).view.render()
    this.addresses().at(x.changedAttributes().selected).view.render()
    CodeBuddy.codeView.render()
  },
  
  // all the existing functions remain  
})

The last thing we need to do is give some way for the user to change the selection. We decided to use jQuery Hotkeys to bind to the up and down arrows (↑ or ↓).

CodeBuddy.setStackKeyBindings = function(){
  $(document).bind('keydown', 'up',   CodeBuddy.stack.selectPrevious)
  $(document).bind('keydown', 'down', CodeBuddy.stack.selectNext)
}

Conclusion

I hope this example has shown a little of the power of using backbone.js when you need to write a complex javascript application.
If you want to see the actual application its all on Github in code_buddy.js

This was our first project working with backbone.js and I’m sure there’s much more we can learn but we’ve found that it lets us write modular code with many small methods which feels much closer to writing Ruby than my previous forays into javascript.

Backbone.js makes writing a javascript application fun!

Backbone.js Makes Building JavaScript Applications Fun

about 1 year ago | Alex Rothenberg: Common Sense Software

Like many developers I’ve had a long, complicated relationship with Javascript. Especially with libraries like jquery it’s incredibly easy to add interesting behavior to your pages, but unless you’re very careful its also likely that you’ll end up with a mess of spaghetti javascript. I know as I’ve gotten myself into that mess and abandoned many projects because they were just too hard to change. All this has changed with some of new libraries out there that help you write your javascript following the MVC pattern.

Today I’m going to talk about backbone.js and show how it helped me and two friends build a rich one-page application to understand an exception stack trace. Backbone.js describes itself as supplying structure to JavaScript-heavy applications by providing models with key-value binding and custom events, collections with a rich API of enumerable functions, views with declarative event handling, and connects it all to your existing application over a RESTful JSON interface.

There are a number of tutorials and examples available out there to get you started. The basic idea is to uses backbone.js to organize your code into models, views and controllers.

What is CodeBuddy

CodeBuddy is the application I’ll describe. It helps you navigate an exception stack raised by your Rails app or any other stack you paste in. I worked on it with Pat Shaughnessy and Daniel Higginbotham and Pat produced a super article describing what CodeBuddy is and how to use it.

There is some server-side code that replaces the Rails Show Exceptions page and syntax highlights a snippet of source code for each line in the stack but for this article I’m going to ignore that and focus on the interactive page and the javascript behind it.

Below is a picture showing a stack trace the background and the code snippet for the currently selected line in front. Now a picture is okay but to really get a sense of it I suggest you follow this link to experience Code Buddy in action - try pressing ↑ or ↓ or double clicking a few lines in the stack then press s.

Code Buddy

There’s a lot going on here and if we had tried building this before discovering backbone.js we probably would have had created a mess - mixing javascript and html that quickly would have become hard to change. I’m going to show you how backbone let us separate the models from the views and create something that was not hard to grow.

Let’s get into the technical details!

Organizing our data in Backbone Models

We want to follow good OO design principles and thinking about this our exception stack is really just a few objects:

  • A Stack

    • has many Addresses
    • knows which Address is selected
  • Each Address has

    • path to a file
    • line number
    • snippet of code

We can build this as a JSON object like

var stackJson = {
  "stack_frames": [
    { "path": "/Users/alex/ruby/github/test_app/app/controllers/users_controller.rb",
      "line": 5,
      "code": "class UsersController < ApplicationController\n..."
    },
    { "path": "/Users/alex/.rvm/.../lib/action_controller/metal/implicit_render.rb",
      "line": 4,
      "code": "module ActionController\n..."
    }
  ],
  "selected": 0
}

The first step is to turn this into backbone models.

// Stack, Address and Addresses
CodeBuddy.backbone.Stack = Backbone.Model.extend({
  initialize: function() {
    this.set({
      addresses: new CodeBuddy.backbone.Addresses(this.get('stack_frames')) 
    })
  }
})

CodeBuddy.backbone.Address = Backbone.Model.extend({
})

CodeBuddy.backbone.Addresses = Backbone.Collection.extend({
  model:CodeBuddy.backbone.Address
})

The way we build a model in backbone is by extending Backbone.Model. When extending we can add custom behavior if we want. In our example we tell the Stack to contain a collection of Address objects in the Addresses collection. We use the backbone collection framework to define Addresses and tell it that it is a collection of Address model objects. These models will use the default backbone behavior for the rest which includes read/write access to its properties get or set.

Now that they’re defined, we’re ready to interact with these models. For example below is what you’d see using the Chrome javascript console (I’m showing the output as comments for readability)

// Using our models in a console
CodeBuddy.stack = new CodeBuddy.backbone.Stack({
  "stack_frames": [
    { "path": "/Users/alex/ruby/github/test_app/app/controllers/users_controller.rb",
      "line": 5,
      "code": "class UsersController < ApplicationController\n..."
    },
    { "path": "/Users/alex/.rvm/.../lib/action_controller/metal/implicit_render.rb",
      "line": 4,
      "code": "module ActionController\n..."
    }
  ],
  "selected": 0
})
// inherits.child

CodeBuddy.stack.get('addresses')
// inherits.child

CodeBuddy.stack.get('addresses').first().get('path')
// "/Users/alex/ruby/github/test_app/app/controllers/users_controller.rb"

CodeBuddy.stack.get('addresses').first().get('code')
// "class UsersController < ApplicationController
// ..."

We now have an object hierarchy with default behavior which we can (and will) extend in a bit but first let’s build some views and get a page we can look at.

Building the UI with Backbone Views

We’re going to build

  • A StackView that contains many AddressViews
  • Each AddressView will display a single line from the stack.
  • Each view will be tied to one of our model objects

It will look something like this:

Code Buddy views on the page

We can start with a StackView that’s tied to the Stack model. Creating a backbone view is very similar to how we created our models - we extend Backbone.View and can override behavior if we want.

CodeBuddy.backbone.StackView = Backbone.View.extend({

  el: $("#stack"),

  initialize: function() {
    this.model.get('addresses').each(this.addOneAddress);
  },
  
  addOneAddress: function(address, index) {
    var view = new CodeBuddy.backbone.AddressView({model: address});
    this.$("#stack").append(view.render().el);
  }
})

In this case we overrode the initializer function to create an AddressView for each address. It also uses jQuery to add the AddressView's html within the page’s #stack element.

To do the iteration we use another powerful javascript library called underscore.js. Underscore.js gives us a ruby-like collection methods letting us write .each(this.addOneAddress). This will iterate over all the addressses calling the addOneAddress function on each one. Underscore.js also gives us erb-like templating we’ll use in the AddressView…let’s take a look at that view.

CodeBuddy.backbone.AddressView = Backbone.View.extend({
  tagName:  "li",

  template: _.template("<span class='container'><%= path %>:<%= line%></span>"),

  initialize: function() {
  },

  render: function() {
    var html = this.template(this.model.toJSON())
    $(this.el).html(html);
    return this;
  }
})

You can see the template and how it does look like erb. It defines the html that will be displayed for each address and its able to access properties in the model like path and line. The template gets applied in the render function with the line this.template(this.model.toJSON()). Finally, tagName is used to wrap this html in an li tag.

Now we can put it all together to see on a page. We can start with a toplevel page that has an element with stack, includes our models and views and tells them to load.

<html>
<body>
  <ul id="stack"></ul>
  <script src="javascripts/code_buddy.js" type="text/javascript"></script>
  <script>
      CodeBuddy.setup({
        "stack_frames": [
          { "path": "/Users/alex/ruby/github/test_app/app/controllers/users_controller.rb",
            "line": 5,
            "code": "class UsersController < ApplicationController\n..."
          },
          { "path": "/Users/alex/.rvm/.../lib/action_controller/metal/implicit_render.rb",
            "line": 4,
            "code": "module ActionController\n..."
          }
        ],
        "selected": 0
      })
  </script>
</body>
</html>

After the page loads and the views render it changes the #stack div to have all this

<ul id="stack">
  <li>
    <span class="container">/Users/alex/ruby/github/test_app/app/controllers/users_controller.rb:5</span>
  </li>
  <li>
    <span class="container">/Users/alex/.rvm/.../lib/action_controller/metal/implicit_render.rb:4</span>
  </li>
</ul>

So far this is a lot of framework and structure for a simple page but now is when it gets interesting and backbone reveals its true power!

Showing code for the selected address

Let’s say we want to mark one address selected and show the code for that one.

Showing Code for selected Addresss

We can start by adding an element to the html called code-viewer

<html>
<body>
  <ul id="stack"></ul>
  <div id="code-viewer">
    <div id="code"></div>
  </div>
  <!-- all the javascript omitted for clarity -->
</body>
</html>

Now we want to create a StackView to put the right code in there

CodeBuddy.backbone.CodeView = Backbone.View.extend({
  el:$("#code-viewer"),
  
  initialize: function() {
  },
  
  render: function() {
    this.$("#code").html(CodeBuddy.stack.selectedAddress().get('code'))
  }
})

In this view we used our models to find which address was selected CodeBuddy.stack.selectedAddress() so let’s add that method.

CodeBuddy.backbone.Stack = Backbone.Model.extend({
  initialize: function() {
    // same code as before
  },

  addresses: function() {
    return this.get('addresses')
  },

  selectedAddress: function() {
    var selected = this.get('selected')
    return this.addresses().at(selected)
  }
})

That’s it, now we have a code view on top of the stack view and our code is still clean and well organized. I’m not afraid to keep going and make our next change.

Changing the selected address

Oh, we just got a new requirement to be able to scroll up and down through the stack and see the code window change as we go.

We add some more functions to our Stack Model so we can change the selection and we also tell it to call the selectionChanged function whenever the selected property changes. Calling a function on a property change is something that’s given to us with backbone events. Getting the view to update itself is also easy we just tell it to with CodeBuddy.codeView.render()

CodeBuddy.backbone.Stack = Backbone.Model.extend({
  initialize: function() {
    this.bind('change:selected', this.selectionChanged);
    // same code as before
  },

  setSelection: function(newSelected) {
    if (newSelected >= 0 && newSelected < this.addresses().size()) {
      this.set({ selected: newSelected })
    }
  },
  
  selectPrevious: function() {
    this.setSelection(this.get('selected') - 1)
  },
  
  selectNext: function() {
    this.setSelection(this.get('selected') + 1)
  }, 

  selectionChanged: function(x) {
    this.addresses().at(x.previousAttributes().selected).view.render()
    this.addresses().at(x.changedAttributes().selected).view.render()
    CodeBuddy.codeView.render()
  },
  
  // all the existing functions remain  
})

The last thing we need to do is give some way for the user to change the selection. We decided to use jQuery Hotkeys to bind to the up and down arrows (↑ or ↓).

CodeBuddy.setStackKeyBindings = function(){
  $(document).bind('keydown', 'up',   CodeBuddy.stack.selectPrevious)
  $(document).bind('keydown', 'down', CodeBuddy.stack.selectNext)
}

Conclusion

I hope this example has shown a little of the power of using backbone.js when you need to write a complex javascript application.
If you want to see the actual application its all on Github in code_buddy.js

This was our first project working with backbone.js and I’m sure there’s much more we can learn but we’ve found that it lets us write modular code with many small methods which feels much closer to writing Ruby than my previous forays into javascript.

Backbone.js makes writing a javascript application fun!

Moved Blog to Jekyll and GitHub Pages

about 1 year ago | Alex Rothenberg: Common Sense Software

I just moved my blog to Jekyll and Github Pages which I hope will help me get back into the rhythm of posting regularly.

Why did I switch?

There were several things I wanted to change

  • Syntax highlighting for my code — Jekyll uses pygments which supports about a bazillion languages including “Gherkin” (cucumber) and blogger had nothing.
  • Authoring tools I wanted — The blogger wysiwyg editor never really worked for me and I’d prefer to use TextMate and markdown or textile anyway.
  • New site design — Making the move gave me a chance to redo my design. The old one felt cluttered and dated and I hope this new one is not.
  • Geek street cred — I like having everything on github and besides all the cool kids are doing it :)

Jekyll is a gem that converts a structured directory of pages and posts written in markdown, textile or html into static html files that can be served by apache or nginx. Of course there’s an official definition that explains it in more detail.

With Github Pages I just have to create a git repository alexrothenberg.github.com and with each push github runs jekyll to generate a static site it serves up at http://alexrothenberg.github.com

How did I create this new site?

1. Create an empty site

There are lots of sites others have created with jekyll to peruse for design ideas. I spent some time looking through them and eventually picked one to use as a template. I copied one and pushed it to a new project on github called alexrothenberg.github.com.

2. Get my existing content

First I had to get my old content out of blogger into a Jekyll site. There are instructions for migrating from many different platforms (including from blogger to jekyll). I followed the instructions to “import with vilcans’ Jekyll rss_importer” and just had to change the rss url to http://www.alexrothenberg.com/feeds/posts/full?alt=rss&max-results=1000 since I had more than 25 posts.

I now had my posts in the _posts folder. I copied these posts into my alexrothenberg.github.com/_posts and had a version of my blog with all my old posts.

3. Syntax highlighting code

I next did some experimentation with the old posts to see how the code highlighting worked. It’s very natural and in-the-flow.

While writing my article I just add a block like

{ % highlight ruby %}
class User
  def name
    [first, last].join(' ') 
  end
end
{ % endhighlight %}

and get this fantastic output

class User
  def name
    [first, last].join(' ') 
  end
end

Well not quite… It generated html with all sorts of css classes but I couldn’t find css stylesheets out there with the Vibrant Ink theme I like.
Through some trial and error I created my own vibrant_ink.css

4. Disqus comments

Because Jekyll generates static pages it has no commenting engine so I have to integrate with an external service. I decided to go with disqus.
Adding it into my site was simple but getting my old comments in turned out to be much harder than it should have been.

First I went to disqus.com, signed up for an account and used their admin too to create a site for www.alexrothenberg.com. They give you a bit of javascript to add to your page. I added this to my _layouts/post.html and I was done.

It looked like it would be just as easy to import my comments from blogger as they have a big import from blogger button on their admin site under Tools->Import/Export. I ran it and I suddenly had all my old comments. But this is where things started to break down. Blogger ties the comments to blogger profiles instead of an email address so none of my comments had avatar pictures next to them.

What followed was a frustrating journey to try and edit the comments using their admin tool. The first thing I tried was using their admin tool but it doesn’t let you edit a commenter’s email or url so I couldn’t stick email addresses in when I knew them (for example I know my own email). My next approach was to export the comments in disqus, edit the xml file it gave me and re-upload. The problem here is that the exported xml uses a different format than the import accepts although they don’t tell you that when you try importing the file you exported it just tells you 0 items imported - how annoying?!?

After a lot of trial and error I wound up with all my comments in an xml file that disqus could import. In case you want to do something similar here are the steps I followed

  1. Create a temp site in disqus (I called mine alexrothenberg1)
  2. Import the comments from blogger using the Disqus import tool
  3. Export those comments to an xml file - I saved mine in alexrothenberg.github.com/_disqus_import/exported_comments.xml
  4. Run my script alexrothenberg.github.com/_disqus_import/exported2import to migrate the format to a Word Press XFR and change the timestamp format.
  5. Hand edit
    alexrothenberg.github.com/_disqus_import/import_comments.xml to update the emails and urls for each comment (luckily I didn’t have that many old comments)
  6. Delete the temp discus site and create a new one
  7. Import my XFR file to the new site
  8. Jekyll and Blogger use different formats for post urls (jekyll includes year/month/date while blogger is year/month) so we need to run the disqus Migrate Threads->Upload a URL Map tool to upload a file like alexrothenberg.github.com/_disqus_import/url.map

5. Mapping my domain name

At this point I have a blog site up at http://alexrothenberg.github.com and its only a quick step to start serving it at http://www.alexrothenberg.com.

  1. Create a CNAME containing www.alexrothenberg.com in my project
  2. Update the dns entry in my domain to create a CNAME record aliasing www to alexrothenberg.github.com

Authoring and Publishing

Now that the site is up and running it was time to write my first article (this one).

  1. jekyll --auto --server

    Starts the jekyll server running at http://localhost:4000 and monitors the files so it regenerates each time I save an edit

  2. newpost Moved Blog to Jekyll and GitHub Pages

    Creates a new empty post using a script I stuck in my project _bin (thanks al3x for this gist)

  3. Start writing content in _posts/2011-01-27-moved-blog-to-jekyll-and-github-pages.md

    Each time I hit save I can preview in a few seconds at http://localhost:4000

  4. git commit -am 'wrote the article'

  5. git push origin master

In a few seconds github processes the site into static files and its published for all the world to see.

Moved Blog to Jekyll and GitHub Pages

about 1 year ago | Alex Rothenberg: Common Sense Software

I just moved my blog to Jekyll and Github Pages which I hope will help me get back into the rhythm of posting regularly.

Why did I switch?

There were several things I wanted to change

  • Syntax highlighting for my code — Jekyll uses pygments which supports about a bazillion languages including “Gherkin” (cucumber) and blogger had nothing.
  • Authoring tools I wanted — The blogger wysiwyg editor never really worked for me and I’d prefer to use TextMate and markdown or textile anyway.
  • New site design — Making the move gave me a chance to redo my design. The old one felt cluttered and dated and I hope this new one is not.
  • Geek street cred — I like having everything on github and besides all the cool kids are doing it :)

Jekyll is a gem that converts a structured directory of pages and posts written in markdown, textile or html into static html files that can be served by apache or nginx. Of course there’s an official definition that explains it in more detail.

With Github Pages I just have to create a git repository alexrothenberg.github.com and with each push github runs jekyll to generate a static site it serves up at http://alexrothenberg.github.com

How did I create this new site?

1. Create an empty site

There are lots of sites others have created with jekyll to peruse for design ideas. I spent some time looking through them and eventually picked one to use as a template. I copied one and pushed it to a new project on github called alexrothenberg.github.com.

2. Get my existing content

First I had to get my old content out of blogger into a Jekyll site. There are instructions for migrating from many different platforms (including from blogger to jekyll). I followed the instructions to “import with vilcans’ Jekyll rss_importer” and just had to change the rss url to http://www.alexrothenberg.com/feeds/posts/full?alt=rss&max-results=1000 since I had more than 25 posts.

I now had my posts in the _posts folder. I copied these posts into my alexrothenberg.github.com/_posts and had a version of my blog with all my old posts.

3. Syntax highlighting code

I next did some experimentation with the old posts to see how the code highlighting worked. It’s very natural and in-the-flow.

While writing my article I just add a block like

{ % highlight ruby %}
class User
  def name
    [first, last].join(' ') 
  end
end
{ % endhighlight %}

and get this fantastic output

class User
  def name
    [first, last].join(' ') 
  end
end

Well not quite… It generated html with all sorts of css classes but I couldn’t find css stylesheets out there with the Vibrant Ink theme I like.
Through some trial and error I created my own vibrant_ink.css

4. Disqus comments

Because Jekyll generates static pages it has no commenting engine so I have to integrate with an external service. I decided to go with disqus.
Adding it into my site was simple but getting my old comments in turned out to be much harder than it should have been.

First I went to disqus.com, signed up for an account and used their admin too to create a site for www.alexrothenberg.com. They give you a bit of javascript to add to your page. I added this to my _layouts/post.html and I was done.

It looked like it would be just as easy to import my comments from blogger as they have a big import from blogger button on their admin site under Tools->Import/Export. I ran it and I suddenly had all my old comments. But this is where things started to break down. Blogger ties the comments to blogger profiles instead of an email address so none of my comments had avatar pictures next to them.

What followed was a frustrating journey to try and edit the comments using their admin tool. The first thing I tried was using their admin tool but it doesn’t let you edit a commenter’s email or url so I couldn’t stick email addresses in when I knew them (for example I know my own email). My next approach was to export the comments in disqus, edit the xml file it gave me and re-upload. The problem here is that the exported xml uses a different format than the import accepts although they don’t tell you that when you try importing the file you exported it just tells you 0 items imported - how annoying?!?

After a lot of trial and error I wound up with all my comments in an xml file that disqus could import. In case you want to do something similar here are the steps I followed

  1. Create a temp site in disqus (I called mine alexrothenberg1)
  2. Import the comments from blogger using the Disqus import tool
  3. Export those comments to an xml file - I saved mine in alexrothenberg.github.com/_disqus_import/exported_comments.xml
  4. Run my script alexrothenberg.github.com/_disqus_import/exported2import to migrate the format to a Word Press XFR and change the timestamp format.
  5. Hand edit
    alexrothenberg.github.com/_disqus_import/import_comments.xml to update the emails and urls for each comment (luckily I didn’t have that many old comments)
  6. Delete the temp discus site and create a new one
  7. Import my XFR file to the new site
  8. Jekyll and Blogger use different formats for post urls (jekyll includes year/month/date while blogger is year/month) so we need to run the disqus Migrate Threads->Upload a URL Map tool to upload a file like alexrothenberg.github.com/_disqus_import/url.map

5. Mapping my domain name

At this point I have a blog site up at http://alexrothenberg.github.com and its only a quick step to start serving it at http://www.alexrothenberg.com.

  1. Create a CNAME containing www.alexrothenberg.com in my project
  2. Update the dns entry in my domain to create a CNAME record aliasing www to alexrothenberg.github.com

Authoring and Publishing

Now that the site is up and running it was time to write my first article (this one).

  1. jekyll --auto --server

    Starts the jekyll server running at http://localhost:4000 and monitors the files so it regenerates each time I save an edit

  2. newpost Moved Blog to Jekyll and GitHub Pages

    Creates a new empty post using a script I stuck in my project _bin (thanks al3x for this gist)

  3. Start writing content in _posts/2011-01-27-moved-blog-to-jekyll-and-github-pages.md

    Each time I hit save I can preview in a few seconds at http://localhost:4000

  4. git commit -am 'wrote the article'

  5. git push origin master

In a few seconds github processes the site into static files and its published for all the world to see.

Setting up a new Mac for Rails Development

about 1 year ago | Alex Rothenberg: Common Sense Software

I just got a new MacBook (whoo hoo!) because my old machine dies (boo hoo) and thought I'd write down the steps I'm taking to set it up. I imagine this is more for myself than anyone else but keep reading if you like.

  • Some Apps
    1. Google Chrome for browsing
    2. TextMate for editing
    3. Adium for IM

  • Homebrew for package management
    1. Open a terminal and type
      ruby -e "$(curl -fsSL https://gist.github.com/raw/323731/install_homebrew.rb)
      
    2. Download and Install XCode from Apple

  • git & github for development
    1. Install GIT
      brew install git
      
    2. create ssh keys and add to github following these instructions
    3. Configure git
      git config --global user.name "Alex Rothenberg"
      git config --global user.email "alex@alexrothenberg.com"
      git config --global color.ui true

  • RVM
    1. Install rvm
    2. add bundler to ~/.rvm/gemsets/global.gems and install it
      echo bundler >> ~/.rvm/gemsets/global.gems
      gem install bundler
    3. install Ruby 1.8.7 and make it default
      rvm install 1.8.7
      rvm --default 1.8.7

  • Databases
    1. MySQL
      brew install mysql
      
    2. Oracle (this is complicated but works if you follow the directions exactly TO THE LETTER without missing a single step!) http://blog.rayapps.com/2009/09/14/how-to-install-oracle-database-10g-on-mac-os-x-snow-leopard/. Then I did have to run these extra commands
      1. Add oracle env settings to my user
        #.bashrc
        ## ORACLE
        export ORACLE_HOME=/Users/oracle/oracle/product/10.2.0/db_1
        export DYLD_LIBRARY_PATH=$ORACLE_HOME/lib
        export ORACLE_SID=orcl
        PATH=$PATH:$ORACLE_HOME/bin

      2. Enable me to access the oracle files
        sudo dscl localhost -append /Local/Default/Groups/oinstall GroupMembership alex
        
      3. SQL Developer is an Oracle GUI

  • Test with a project
    git clone git://github.com/alexrothenberg/legacy_data.git
    cd legacy_data
    export MYSQL
    export ORACLE
    bundle install
    rake
    

  • Finally I'm following the trend and setting up a dotfiles project on github so I don't have to recreate them in the future.

Setting up a new Mac for Rails Development

about 1 year ago | Alex Rothenberg: Common Sense Software

I just got a new MacBook (whoo hoo!) because my old machine dies (boo hoo) and thought I'd write down the steps I'm taking to set it up. I imagine this is more for myself than anyone else but keep reading if you like.

  • Some Apps
    1. Google Chrome for browsing
    2. TextMate for editing
    3. Adium for IM

  • Homebrew for package management
    1. Open a terminal and type
      ruby -e "$(curl -fsSL https://gist.github.com/raw/323731/install_homebrew.rb)
      
    2. Download and Install XCode from Apple

  • git & github for development
    1. Install GIT
      brew install git
      
    2. create ssh keys and add to github following these instructions
    3. Configure git
      git config --global user.name "Alex Rothenberg"
      git config --global user.email "alex@alexrothenberg.com"
      git config --global color.ui true

  • RVM
    1. Install rvm
    2. add bundler to ~/.rvm/gemsets/global.gems and install it
      echo bundler >> ~/.rvm/gemsets/global.gems
      gem install bundler
    3. install Ruby 1.8.7 and make it default
      rvm install 1.8.7
      rvm --default 1.8.7

  • Databases
    1. MySQL
      brew install mysql
      
    2. Oracle (this is complicated but works if you follow the directions exactly TO THE LETTER without missing a single step!) http://blog.rayapps.com/2009/09/14/how-to-install-oracle-database-10g-on-mac-os-x-snow-leopard/. Then I did have to run these extra commands
      1. Add oracle env settings to my user
        #.bashrc
        ## ORACLE
        export ORACLE_HOME=/Users/oracle/oracle/product/10.2.0/db_1
        export DYLD_LIBRARY_PATH=$ORACLE_HOME/lib
        export ORACLE_SID=orcl
        PATH=$PATH:$ORACLE_HOME/bin

      2. Enable me to access the oracle files
        sudo dscl localhost -append /Local/Default/Groups/oinstall GroupMembership alex
        
      3. SQL Developer is an Oracle GUI

  • Test with a project
    git clone git://github.com/alexrothenberg/legacy_data.git
    cd legacy_data
    export MYSQL
    export ORACLE
    bundle install
    rake
    

  • Finally I'm following the trend and setting up a dotfiles project on github so I don't have to recreate them in the future.

Combining two related fields in a url to prevent tampering

over 1 year ago | Alex Rothenberg: Common Sense Software

Usually when you select an item from a search results list you only need to pass the id in the url and that's enough to lookup the rest of the details when processing the response. Recently I had a case where I couldn't lookup the item from the id because it wasn't in my database. I'm going to describe how I passed several pieces of information in a single url and what I did to ensure users could not be manipulate them independently.

My application was searching and displaying a list of companies then keeping track of which one the user selects.



When someone clicks on a company name it would save the company's id and display a nice message "Thanks for selecting Google Inc." Since the companies came from a slow external service I didn't want to call again, I passed both the id and name in the url.

I built a couple of simple haml files to show my company list. Each row had a link to display the company name and passed both the name and id in the url.

# search_results.html.haml
%table
  %tr
    %th Company Name
    %th Location
  = render @companies


# _company.html.haml partial
%tr
  %td= link_to company.name, company_path(:id=>company.id, :name=>company.name)
  %td= company.location



The controller code was also pretty straightforward.

class CompaniesController
  def show
    company = Company.new(:id=>params[:id], :name => params[:name])
    current_user.update_selected_company = company
  end
end



This worked great and worked, but then we started looking at the urls in our browser and noticed they looked like http://my.site.com/companies/60902413&name=Google+Inc..

Hmm ... What would happen if someone changed the name in the url? We tried loading http://my.site.com/companies/60902413&name=Some+Silly+Name. Uh-oh, our database now stored the selected company Some Silly Name with id #60902413. This could be confusing or worse a security risk where a clever (or malicious) user could store inaccurate information in our database.

Our solution was to combine the two fields into a single query parameter that was resistant to user tampering and is passed as a single unit. Fortunately Rails passes the session back and forth in a cookie with just that requirement. The session is a hash of many different key-value pairs that need to be encoded as a single cookie, it also contains sensitive information that should be resistant to tampering (and unreadable) and, most importantly it turns out to be something we could reuse.

Starting from the outside in what we wanted to do was rewrite our _company.html.haml partial view to put the single encrypted form of the company on the url.

%tr
  %td= link_to company.name, company_path(:id=>company.to_encrypted_s)
  %td= company.location

and parse that in the controller

class CompaniesController
  def action
    company = Company.from_encrypted_s(params[:id])
    current_user.update_selected_company = company
  end
end



Ok, but what do those to_encrypted_s and from_encrypted_s methods do?

class Company
  def to_encrypted_s
    Encryption.new.encrypt(:company_name=>company_name, :duns=>duns)
  end

  def self.from_encrypted_s encrypted_data
    Company.new Encryption.new.decrypt(encrypted_data)
  end
end

I still haven't told you how they work, we're just working down from the outside-in figuring out what other classes we'll need. So what does the Encryption class look like?

class Encryption
  def initialize
    secret = (Rails.version > '3.0') ?
             Rails.application.config.cookie_secret : 
             ActionController::Base.session_options[:secret]
    @verifier = ActiveSupport::MessageVerifier.new(secret, 'SHA1')
  end

  def encrypt message
    @verifier.generate message
  end

  def decrypt encrypted
    @verifier.verify encrypted
  end
end



We make use of the ActiveSupport::MessageVerifier class which as the documentation says "makes it easy to generate and verify messages which are signed to prevent tampering".

Now when we select a company we get a url with a crazy long id like http://my.site.com/company/BAh7BzoJZHVuc2kDE%2BAUOhFjb21wYW55X25hbWUiMEludGVybmF0aW9uYWwgQnVzaW5lc3MgTWFjaGluZXMgQ29ycG9yYXRpb24%3D--9027b2c449c0b4a1aea375cb1722fa9db8e56066. If someone were to try and change the really long id in the url the application will throw an ActiveSupport::MessageVerifier::InvalidSignature exception instead of saving bad data.

We've given up readability in the url but in exchange we're guaranteed the id and name we get in the controller go with each other.

Combining two related fields in a url to prevent tampering

over 1 year ago | Alex Rothenberg: Common Sense Software

Usually when you select an item from a search results list you only need to pass the id in the url and that's enough to lookup the rest of the details when processing the response. Recently I had a case where I couldn't lookup the item from the id because it wasn't in my database. I'm going to describe how I passed several pieces of information in a single url and what I did to ensure users could not be manipulate them independently.

My application was searching and displaying a list of companies then keeping track of which one the user selects.



When someone clicks on a company name it would save the company's id and display a nice message "Thanks for selecting Google Inc." Since the companies came from a slow external service I didn't want to call again, I passed both the id and name in the url.

I built a couple of simple haml files to show my company list. Each row had a link to display the company name and passed both the name and id in the url.

# search_results.html.haml
%table
  %tr
    %th Company Name
    %th Location
  = render @companies


# _company.html.haml partial
%tr
  %td= link_to company.name, company_path(:id=>company.id, :name=>company.name)
  %td= company.location



The controller code was also pretty straightforward.

class CompaniesController
  def show
    company = Company.new(:id=>params[:id], :name => params[:name])
    current_user.update_selected_company = company
  end
end



This worked great and worked, but then we started looking at the urls in our browser and noticed they looked like http://my.site.com/companies/60902413&name=Google+Inc..

Hmm ... What would happen if someone changed the name in the url? We tried loading http://my.site.com/companies/60902413&name=Some+Silly+Name. Uh-oh, our database now stored the selected company Some Silly Name with id #60902413. This could be confusing or worse a security risk where a clever (or malicious) user could store inaccurate information in our database.

Our solution was to combine the two fields into a single query parameter that was resistant to user tampering and is passed as a single unit. Fortunately Rails passes the session back and forth in a cookie with just that requirement. The session is a hash of many different key-value pairs that need to be encoded as a single cookie, it also contains sensitive information that should be resistant to tampering (and unreadable) and, most importantly it turns out to be something we could reuse.

Starting from the outside in what we wanted to do was rewrite our _company.html.haml partial view to put the single encrypted form of the company on the url.

%tr
  %td= link_to company.name, company_path(:id=>company.to_encrypted_s)
  %td= company.location

and parse that in the controller

class CompaniesController
  def action
    company = Company.from_encrypted_s(params[:id])
    current_user.update_selected_company = company
  end
end



Ok, but what do those to_encrypted_s and from_encrypted_s methods do?

class Company
  def to_encrypted_s
    Encryption.new.encrypt(:company_name=>company_name, :duns=>duns)
  end

  def self.from_encrypted_s encrypted_data
    Company.new Encryption.new.decrypt(encrypted_data)
  end
end

I still haven't told you how they work, we're just working down from the outside-in figuring out what other classes we'll need. So what does the Encryption class look like?

class Encryption
  def initialize
    secret = (Rails.version > '3.0') ?
             Rails.application.config.cookie_secret : 
             ActionController::Base.session_options[:secret]
    @verifier = ActiveSupport::MessageVerifier.new(secret, 'SHA1')
  end

  def encrypt message
    @verifier.generate message
  end

  def decrypt encrypted
    @verifier.verify encrypted
  end
end



We make use of the ActiveSupport::MessageVerifier class which as the documentation says "makes it easy to generate and verify messages which are signed to prevent tampering".

Now when we select a company we get a url with a crazy long id like http://my.site.com/company/BAh7BzoJZHVuc2kDE%2BAUOhFjb21wYW55X25hbWUiMEludGVybmF0aW9uYWwgQnVzaW5lc3MgTWFjaGluZXMgQ29ycG9yYXRpb24%3D--9027b2c449c0b4a1aea375cb1722fa9db8e56066. If someone were to try and change the really long id in the url the application will throw an ActiveSupport::MessageVerifier::InvalidSignature exception instead of saving bad data.

We've given up readability in the url but in exchange we're guaranteed the id and name we get in the controller go with each other.

Purging old cruise builds

over 1 year ago | Alex Rothenberg: Common Sense Software

Continuous Integration is such an important idea and I love CruiseControl.rb as its so easy to use. However one thing that drives me crazy is that it keeps all builds forever. Over the past few years we run out of disk space every few months causing all our builds to start failing and some tedious manual work to delete some directories.

Today I decided to do something about it and wrote a script to remove all but the most recent builds. After all why do I want an old build lying around as that's what Git of Subversion is for!

If you want to give it a try you can download the script from http://github.com/alexrothenberg/purge_old_cruise_builds

This seems to be working for me and I'd love to hear if I'm making some assumptions you need to change to adapt it to your uses.

Purging old cruise builds

over 1 year ago | Alex Rothenberg: Common Sense Software

Continuous Integration is such an important idea and I love CruiseControl.rb as its so easy to use. However one thing that drives me crazy is that it keeps all builds forever. Over the past few years we run out of disk space every few months causing all our builds to start failing and some tedious manual work to delete some directories.

Today I decided to do something about it and wrote a script to remove all but the most recent builds. After all why do I want an old build lying around as that's what Git of Subversion is for!

If you want to give it a try you can download the script from http://github.com/alexrothenberg/purge_old_cruise_builds

This seems to be working for me and I'd love to hear if I'm making some assumptions you need to change to adapt it to your uses.

Use dashes not underscores in your domain name

over 1 year ago | Alex Rothenberg: Common Sense Software

I recently ran into an issue where users hitting my development server with Internet Explorer kept getting redirected back to my login page no matter how many times they signed in. Everything worked fine for all other browsers I tested and it even worked fine on Internet Explorer when pointing at http://localhost.

After much investigating it turns out the problem lies in the name of my development server. I picked a name like http://myapp_dev.alexrothenberg.com and it was the underscore in myapp_dev that caused the problem. It turns out that IE will not set a session cookie for a domain with a hostname that contains an underscore.

According to microsoft this is not a bug but a feature

Q5: IE won’t set a cookie when the hostname/domain contains an underscore?

A: Correct. Technically, underscore is not a DNS character, and while Windows will let you use an underscore when naming your machine, it warns you that doing so may cause problems. One such problem is that WinINET blocks attempts to set cookies on such domains.


OK, so it looks like IE will let me go to the page but doesn't set the session cookie so my site sees each request as part of a new session by an unauthenticated user and redirects to the login page over and over.

I did a little more research and according to Wikipedia Microsoft is right and underscores are not allowed in hostnames

The Internet standards (Request for Comments) for protocols mandate that component hostname labels may contain only the ASCII letters 'a' through 'z' (in a case-insensitive manner), the digits '0' through '9', and the hyphen ('-') ...
While a hostname may not contain other characters, such as the underscore character (_), other DNS names may contain the underscore.


I'm not sure why IE allows the browsing but not the cookie as that turned this into a subtle bug that was especially hard to track down since Firefox, Safari and Chrome all set the cookie so my site works. From now on I'll have to remember not to use underscores but to use dashes instead like http://myapp-dev.alexrothenberg.com which should work across all browsers.

Use dashes not underscores in your domain name

over 1 year ago | Alex Rothenberg: Common Sense Software

I recently ran into an issue where users hitting my development server with Internet Explorer kept getting redirected back to my login page no matter how many times they signed in. Everything worked fine for all other browsers I tested and it even worked fine on Internet Explorer when pointing at http://localhost.

After much investigating it turns out the problem lies in the name of my development server. I picked a name like http://myapp_dev.alexrothenberg.com and it was the underscore in myapp_dev that caused the problem. It turns out that IE will not set a session cookie for a domain with a hostname that contains an underscore.

According to microsoft this is not a bug but a feature

Q5: IE won’t set a cookie when the hostname/domain contains an underscore?

A: Correct. Technically, underscore is not a DNS character, and while Windows will let you use an underscore when naming your machine, it warns you that doing so may cause problems. One such problem is that WinINET blocks attempts to set cookies on such domains.


OK, so it looks like IE will let me go to the page but doesn't set the session cookie so my site sees each request as part of a new session by an unauthenticated user and redirects to the login page over and over.

I did a little more research and according to Wikipedia Microsoft is right and underscores are not allowed in hostnames

The Internet standards (Request for Comments) for protocols mandate that component hostname labels may contain only the ASCII letters 'a' through 'z' (in a case-insensitive manner), the digits '0' through '9', and the hyphen ('-') ...
While a hostname may not contain other characters, such as the underscore character (_), other DNS names may contain the underscore.


I'm not sure why IE allows the browsing but not the cookie as that turned this into a subtle bug that was especially hard to track down since Firefox, Safari and Chrome all set the cookie so my site works. From now on I'll have to remember not to use underscores but to use dashes instead like http://myapp-dev.alexrothenberg.com which should work across all browsers.

Its not too late to solve a Y2K bug - I found one in 2010

almost 2 years ago | Alex Rothenberg: Common Sense Software

I thought a historical opportunity had passed me by, but a full decade after the turn of the millennium in 2010 I found my first Y2K bug! If you, like me, were working in eCommerce (who remembers that buzzword?) in the late 1990s and thought you missed your chance I'm here to tell you its still not too late. As long as you work for the right company with the enough legacy technology around you can still find a Y2K bug of your own!

What was going on?

I was writing a new Rails application on top of an old Oracle database. The database had a constraint


graduationdate BETWEEN TO_DATE('01-JAN-1900','DD-MON-YYYY') AND TO_DATE('31-DEC-2100','DD-MON-YYYY')



To me this looked like a basic check to make sure the date was within a 300 year range as a basic sanity-check. I implemented an ActiveRecord validation in my model

validate :validate_graduation_date
def validate_graduation_date
  unless graduationdate.nil?
    earliest_date, latest_date = Date.parse('01-JAN-1900'), Date.parse('31-DEC-2100')
    errors.add(:graduationdate, 'must be between 01-JAN-1900 and 31-DEC-2100') unless graduationdate > earliest_date and graduationdate < latest_date  
  end
end



Of course I had unit tests that all passed

describe 'graduation date' do
  it 'should not allow July 4, 1776' do
    education = Factory.build :education, :graduationdate=>Date.parse('04-JUL-1776')
    education.should_not be_valid
  end
  it 'should allow Aug 1 1970' do
    education = Factory.build :education, :graduationdate=>Date.parse('01-AUG-1970')
    education.should be_valid
  end
  it 'should allow Mar 15 2000' do
    education = Factory.build :education, :graduationdate=>Date.parse('15-MAR-2000')
    education.should be_valid
  end
  it 'should not allow Jan 1, 2525' do
    education = Factory.build :education, :graduationdate=>Date.parse('01-JAN-2525')
    education.should_not be_valid
  end
end



BUT once I went into production with my legacy database I started getting errors. So I took a look at one of the rows giving me an error and discovered the graduation date was 01-JAN-00. AHA a 2-digit year. That doesn't look good and there were over 100,000 rows with this problem.

select count(*) from education where graduationdate = to_date('01-jan-1900')
COUNT(*)               
---------------------- 
116232       



Once I realized that the database contained data for January 1st 1900 and I discovered the Oracle BETWEEN function is inclusive I was able to solve my problem by changing my ActiveRecord validation to have a >= and lt;=.

validate :validate_graduation_date
def validate_graduation_date
  unless graduationdate.nil?
    earliest_date, latest_date = Date.parse('01-JAN-1900'), Date.parse('31-DEC-2100')
    errors.add(:graduationdate, 'must be between 01-JAN-1900 and 31-DEC-2100') unless graduationdate >= earliest_date and graduationdate <= latest_date  
  end
end



I added some specs to test my boundary conditions

it 'should not allow Dec 31 1899' do
  education = Factory.build :education, :graduationdate=>Date.parse('31-DEC-1899')
  education.should_not be_valid
end
it 'should allow Jan 1 1900' do
  education = Factory.build :education, :graduationdate=>Date.parse('01-JAN-1900')
  education.should be_valid
end
it 'should allow Dec 31 2099' do
  education = Factory.build :education, :graduationdate=>Date.parse('31-DEC-2099')
  education.should be_valid
end
it 'should allow Jan 1 2100' do
  education = Factory.build :education, :graduationdate=>Date.parse('01-JAN-2100')
  education.should_not be_valid
end



So what's the moral?

I'm not sure but when you're have to integrate with a legacy system you should account for the absurd - in my case a job application from a candidate who graduated more than 100 years ago!

And if you feel like you missed out on a once-in-a-lifetime opportunity to work on Y2K bugs don't despair. They are still out there!

Its not too late to solve a Y2K bug - I found one in 2010

almost 2 years ago | Alex Rothenberg: Common Sense Software

I thought a historical opportunity had passed me by, but a full decade after the turn of the millennium in 2010 I found my first Y2K bug! If you, like me, were working in eCommerce (who remembers that buzzword?) in the late 1990s and thought you missed your chance I'm here to tell you its still not too late. As long as you work for the right company with the enough legacy technology around you can still find a Y2K bug of your own!

What was going on?

I was writing a new Rails application on top of an old Oracle database. The database had a constraint


graduationdate BETWEEN TO_DATE('01-JAN-1900','DD-MON-YYYY') AND TO_DATE('31-DEC-2100','DD-MON-YYYY')



To me this looked like a basic check to make sure the date was within a 300 year range as a basic sanity-check. I implemented an ActiveRecord validation in my model

validate :validate_graduation_date
def validate_graduation_date
  unless graduationdate.nil?
    earliest_date, latest_date = Date.parse('01-JAN-1900'), Date.parse('31-DEC-2100')
    errors.add(:graduationdate, 'must be between 01-JAN-1900 and 31-DEC-2100') unless graduationdate > earliest_date and graduationdate < latest_date  
  end
end



Of course I had unit tests that all passed

describe 'graduation date' do
  it 'should not allow July 4, 1776' do
    education = Factory.build :education, :graduationdate=>Date.parse('04-JUL-1776')
    education.should_not be_valid
  end
  it 'should allow Aug 1 1970' do
    education = Factory.build :education, :graduationdate=>Date.parse('01-AUG-1970')
    education.should be_valid
  end
  it 'should allow Mar 15 2000' do
    education = Factory.build :education, :graduationdate=>Date.parse('15-MAR-2000')
    education.should be_valid
  end
  it 'should not allow Jan 1, 2525' do
    education = Factory.build :education, :graduationdate=>Date.parse('01-JAN-2525')
    education.should_not be_valid
  end
end



BUT once I went into production with my legacy database I started getting errors. So I took a look at one of the rows giving me an error and discovered the graduation date was 01-JAN-00. AHA a 2-digit year. That doesn't look good and there were over 100,000 rows with this problem.

select count(*) from education where graduationdate = to_date('01-jan-1900')
COUNT(*)               
---------------------- 
116232       



Once I realized that the database contained data for January 1st 1900 and I discovered the Oracle BETWEEN function is inclusive I was able to solve my problem by changing my ActiveRecord validation to have a >= and lt;=.

validate :validate_graduation_date
def validate_graduation_date
  unless graduationdate.nil?
    earliest_date, latest_date = Date.parse('01-JAN-1900'), Date.parse('31-DEC-2100')
    errors.add(:graduationdate, 'must be between 01-JAN-1900 and 31-DEC-2100') unless graduationdate >= earliest_date and graduationdate <= latest_date  
  end
end



I added some specs to test my boundary conditions

it 'should not allow Dec 31 1899' do
  education = Factory.build :education, :graduationdate=>Date.parse('31-DEC-1899')
  education.should_not be_valid
end
it 'should allow Jan 1 1900' do
  education = Factory.build :education, :graduationdate=>Date.parse('01-JAN-1900')
  education.should be_valid
end
it 'should allow Dec 31 2099' do
  education = Factory.build :education, :graduationdate=>Date.parse('31-DEC-2099')
  education.should be_valid
end
it 'should allow Jan 1 2100' do
  education = Factory.build :education, :graduationdate=>Date.parse('01-JAN-2100')
  education.should_not be_valid
end



So what's the moral?

I'm not sure but when you're have to integrate with a legacy system you should account for the absurd - in my case a job application from a candidate who graduated more than 100 years ago!

And if you feel like you missed out on a once-in-a-lifetime opportunity to work on Y2K bugs don't despair. They are still out there!

What to do when ActiveRecord thinks an Oracle key is a decimal

about 2 years ago | Alex Rothenberg: Common Sense Software

I recently created a model for an existing database table using the legacy_data gem and was confused when my primary key showed up in scientific notation. It turned out the issue was due to sloppiness in the table definition and could be easily fixed once I understood what ActiveRecord was doing.

I created a Person model connected to the people table

class Person < ActiveRecord::Base
end



but when I went into script/console the primary key showed up as a BigDecimal when I expected an integer.

$ script/console 
Loading development environment (Rails 2.3.4)
>> Person.first.id
=> #



This wasn't what I wanted and would cause problems in my app when it tried to build a url with that id like http://localhost:3000/people/10024844425.0. The rails routing engine would see the .0, treat it as a format (like .xml or .json) and get confused. Let's look at why this is happening.


>> Person.columns_hash['id']
=> #<ActiveRecord::ConnectionAdapters::OracleEnhancedColumn:0x37391c0 @default=nil, @type=:decimal,
   #   @null=true, @name="id", @table_name="people", @scale=nil, @sql_type="NUMBER", @precision=nil,
   #   @primary=true, @forced_column_type=nil, @limit=nil>
>> Person.columns_hash['id'].type
=> :decimal
>> Person.columns_hash['id'].sql_type
=> "NUMBER"

We see that ActiveRecord is treating this column as a :decimal because it's sql_type is NUMBER. It turns out this is correct because an Oracle number is a decimal unless you specify it to have 0 digits after the decimal point (scale of 0). Here's the documentation from Oracle (the last sentence is my bold)

NUMBER Datatype

The NUMBER datatype stores fixed and floating-point numbers. Numbers of virtually any magnitude can be stored and are
guaranteed portable among different systems operating Oracle, up to 38 digits of precision.

The following numbers can be stored in a NUMBER column:
  * Positive numbers in the range 1 x 10-130 to 9.99...9 x 10125 with up to 38 significant digits
  * Negative numbers from -1 x 10-130 to 9.99...99 x 10125 with up to 38 significant digits
  * Zero
  * Positive and negative infinity (generated only by importing from an Oracle Version 5 database)

For numeric columns, you can specify the column as:
  column_name NUMBER 

Optionally, you can also specify a precision (total number of digits) and scale (number of digits to the right of the decimal point):
  column_name NUMBER (precision, scale) 

If a precision is not specified, the column stores values as given. If no scale is specified, the scale is zero.

Oracle guarantees portability of numbers with a precision equal to or less than 38 digits. You can specify a scale and no precision:
  column_name NUMBER (*, scale) 

In this case, the precision is 38, and the specified scale is maintained.

When you specify numeric fields, it is a good idea to specify the precision and scale. This provides extra integrity checking on input.



Let's look in my database and sure enough the ID is a number

$ sqlplus myusername/mypassword@localhost:1521/mydatabase.world
SQL*Plus: Release 10.2.0.4.0 - Production on Wed Jan 27 09:15:09 2010
Copyright (c) 1982, 2007, Oracle.  All Rights Reserved.

Connected to:
Oracle Database 10g Release 10.2.0.4.0 - Production
SQL> desc people;
Name        Null?    Type
----------------------------------------- -------- ----------------------------
ID                   NUMBER
NAME                 VARCHAR2(10)



If you are allowed to change your database you can create a migration like


$ script/generate migration change_person_id_to_integer
STUBBING MckinseyLDAP
exists db/migrate
create db/migrate/20100127145747_change_person_id_to_integer.rb



now edit the migration

class ChangePersonIdToInteger < ActiveRecord::Migration
  def self.up
    change_column(:people, :id, :integer)  
  end
  def self.down
    change_column(:people, :id, :decimal)  
  end
end



In my case there were other applications using this table and I was not allowed to change it so I implemented a fix in Ruby to tell my model to treat this column as an integer even though it was defined as a decimal in the database.

#config/initializers/legacy_data_type_coercion.rb
module LegacyDataTypeCoercion
  def set_integer_columns *col_names
    col_names.each do |col_name|
      columns_hash[col_name.to_s].instance_eval do
        @type = :integer
      end
    end
  end
end
ActiveRecord::Base.extend(LegacyDataTypeCoercion)


#app/models/person.rb
class Person < ActiveRecord::Base
  set_integer_columns :id
end



We defined a method set_integer_columns that will force ActiveRecord to treat the columns we specify as integers. In our Person model we declare :id is an integer column. Let's test it out!

$ script/console 
Loading development environment (Rails 2.3.4)
>> Person.first.id
=> 10024844425
>> Person.columns_hash['id'].type
=> :integer



Just as expected id is now an integer and we can go ahead building the rest of our application.

This is not an issue with all Oracle tables as if the column was defined as NUMBER(10) (with a precision and implicit scale of 0) then ActiveRecord will interpret it as an integer automatically based on the parentheses in the data type - i.e NUMBER(10) ActiveRecord or Oracle Enhanced).

What to do when ActiveRecord thinks an Oracle key is a decimal

about 2 years ago | Alex Rothenberg: Common Sense Software

I recently created a model for an existing database table using the legacy_data gem and was confused when my primary key showed up in scientific notation. It turned out the issue was due to sloppiness in the table definition and could be easily fixed once I understood what ActiveRecord was doing.

I created a Person model connected to the people table

class Person < ActiveRecord::Base
end



but when I went into script/console the primary key showed up as a BigDecimal when I expected an integer.

$ script/console 
Loading development environment (Rails 2.3.4)
>> Person.first.id
=> #



This wasn't what I wanted and would cause problems in my app when it tried to build a url with that id like http://localhost:3000/people/10024844425.0. The rails routing engine would see the .0, treat it as a format (like .xml or .json) and get confused. Let's look at why this is happening.


>> Person.columns_hash['id']
=> #<ActiveRecord::ConnectionAdapters::OracleEnhancedColumn:0x37391c0 @default=nil, @type=:decimal,
   #   @null=true, @name="id", @table_name="people", @scale=nil, @sql_type="NUMBER", @precision=nil,
   #   @primary=true, @forced_column_type=nil, @limit=nil>
>> Person.columns_hash['id'].type
=> :decimal
>> Person.columns_hash['id'].sql_type
=> "NUMBER"

We see that ActiveRecord is treating this column as a :decimal because it's sql_type is NUMBER. It turns out this is correct because an Oracle number is a decimal unless you specify it to have 0 digits after the decimal point (scale of 0). Here's the documentation from Oracle (the last sentence is my bold)

NUMBER Datatype

The NUMBER datatype stores fixed and floating-point numbers. Numbers of virtually any magnitude can be stored and are
guaranteed portable among different systems operating Oracle, up to 38 digits of precision.

The following numbers can be stored in a NUMBER column:
  * Positive numbers in the range 1 x 10-130 to 9.99...9 x 10125 with up to 38 significant digits
  * Negative numbers from -1 x 10-130 to 9.99...99 x 10125 with up to 38 significant digits
  * Zero
  * Positive and negative infinity (generated only by importing from an Oracle Version 5 database)

For numeric columns, you can specify the column as:
  column_name NUMBER 

Optionally, you can also specify a precision (total number of digits) and scale (number of digits to the right of the decimal point):
  column_name NUMBER (precision, scale) 

If a precision is not specified, the column stores values as given. If no scale is specified, the scale is zero.

Oracle guarantees portability of numbers with a precision equal to or less than 38 digits. You can specify a scale and no precision:
  column_name NUMBER (*, scale) 

In this case, the precision is 38, and the specified scale is maintained.

When you specify numeric fields, it is a good idea to specify the precision and scale. This provides extra integrity checking on input.



Let's look in my database and sure enough the ID is a number

$ sqlplus myusername/mypassword@localhost:1521/mydatabase.world
SQL*Plus: Release 10.2.0.4.0 - Production on Wed Jan 27 09:15:09 2010
Copyright (c) 1982, 2007, Oracle.  All Rights Reserved.

Connected to:
Oracle Database 10g Release 10.2.0.4.0 - Production
SQL> desc people;
Name        Null?    Type
----------------------------------------- -------- ----------------------------
ID                   NUMBER
NAME                 VARCHAR2(10)



If you are allowed to change your database you can create a migration like


$ script/generate migration change_person_id_to_integer
STUBBING MckinseyLDAP
exists db/migrate
create db/migrate/20100127145747_change_person_id_to_integer.rb



now edit the migration

class ChangePersonIdToInteger < ActiveRecord::Migration
  def self.up
    change_column(:people, :id, :integer)  
  end
  def self.down
    change_column(:people, :id, :decimal)  
  end
end



In my case there were other applications using this table and I was not allowed to change it so I implemented a fix in Ruby to tell my model to treat this column as an integer even though it was defined as a decimal in the database.

#config/initializers/legacy_data_type_coercion.rb
module LegacyDataTypeCoercion
  def set_integer_columns *col_names
    col_names.each do |col_name|
      columns_hash[col_name.to_s].instance_eval do
        @type = :integer
      end
    end
  end
end
ActiveRecord::Base.extend(LegacyDataTypeCoercion)


#app/models/person.rb
class Person < ActiveRecord::Base
  set_integer_columns :id
end



We defined a method set_integer_columns that will force ActiveRecord to treat the columns we specify as integers. In our Person model we declare :id is an integer column. Let's test it out!

$ script/console 
Loading development environment (Rails 2.3.4)
>> Person.first.id
=> 10024844425
>> Person.columns_hash['id'].type
=> :integer



Just as expected id is now an integer and we can go ahead building the rest of our application.

This is not an issue with all Oracle tables as if the column was defined as NUMBER(10) (with a precision and implicit scale of 0) then ActiveRecord will interpret it as an integer automatically based on the parentheses in the data type - i.e NUMBER(10) ActiveRecord or Oracle Enhanced).

Freezing a gem that has native extensions

about 2 years ago | Alex Rothenberg: Common Sense Software

I like to freeze all the gems I use as we run in a shared hosting environment and need to our apps isolated from each other. Deployments are also handled by an operational team that does not intimately understand our applications so keeping our deployments to a single capistrano command cap deploy:migrations has been a big win for us. Freezing most gems is pretty straightforward and has been built in since Rails 2.1. When dealing with a gem that requires native extensions to be built there's only one additional step to add to your Capfil.

Let's say we want to localize hpricot which does include native C extensions.

First tell Rails about your gem by adding a config.gem line to your environment.rb


Rails::Initializer.run do |config|
...
config.gem 'hpricot'
...
end



Now we can ask rails about its configured gems


$ rake gems
(in /Users/alexrothenberg/ruby/my_project)
- [I] hpricot

I = Installed
F = Frozen
R = Framework (loaded before rails starts)



The 'I' means its hpricot is installed on my system but not frozen in the application. If you see '[]' instead you need to run sudo gem install hpricot (add '--source http://gemcutter.org' if necessary). At this point you could write some code to use hpricot and your application will work. But if hpricot (or the version you're expecting) is not installed on your production server you'll be in trouble.

To freeze the gem into your vendor directory run rake gems:unpack (optionally you can add 'GEM=hpricot' if you just want to unpack one gem).


$ rake gems:unpack
(in /Users/alexrothenberg/ruby/my_project)
Unpacked gem: '/Users/alexrothenberg/ruby/my_project/vendor/gems/hpricot-0.8.2'



We can ask rails again...


$ rake gems
(in /Users/alexrothenberg/ruby/my_project)
The following gems have native components that need to be built
hpricot

You're running:
ruby 1.8.6.287 at /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby
rubygems 1.3.2 at /Users/alexrothenberg/.gem/ruby/1.8, /Library/Ruby/Gems/1.8, /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/gems/1.8

Run `rake gems:build` to build the unbuilt gems.



Oops our vendored gem is missing hasn't built the native extensions. Not to worry the message tells us what to do and we run rake gems:build


$ rake gems:build
(in /Users/alexrothenberg/ruby/my_project)
Built gem: '/Users/alexrothenberg/ruby/mars-admin/vendor/gems/hpricot-0.8.2'
alex-rothenbergs:mars-admin alexrothenberg$ rake gems
(in /Users/alexrothenberg/ruby/my_project)
- [F] hpricot

I = Installed
F = Frozen
R = Framework (loaded before rails starts)



We can ask rails again to see that the gem is now frozen and also look in our vendor folder


$ rake gems
(in /Users/alexrothenberg/ruby/my_project)
- [F] hpricot

I = Installed
F = Frozen
R = Framework (loaded before rails starts)

$ ls vendor/gems/hpricot-0.8.2/
total 72
-rw-r--r-- 1 alexrothenberg staff 4672 Jan 13 12:33 CHANGELOG
-rw-r--r-- 1 alexrothenberg staff 1048 Jan 13 12:33 COPYING
-rw-r--r-- 1 alexrothenberg staff 9216 Jan 13 12:33 README
-rw-r--r-- 1 alexrothenberg staff 8242 Jan 13 12:33 Rakefile
drwxr-xr-x 4 alexrothenberg staff 136 Jan 13 12:33 ext/
drwxr-xr-x 3 alexrothenberg staff 102 Jan 13 12:33 extras/
drwxr-xr-x 6 alexrothenberg staff 204 Jan 13 12:39 lib/
drwxr-xr-x 11 alexrothenberg staff 374 Jan 13 12:33 test/



Everything looks good and you can check this into git and now have a frozen version of the hpricot gem stored with your application.
But if we stop here, when we deploy to our production server we'd be using the native extensions we built on your laptop which may not work on the server if you have one is 32bit and the other 64bit or you have different OS libraries installed or any number of other reasons.

To be safe, we need to rebuild the native extensions on the server when we deploy. This is not as hard as it sounds as rails gave us the rake task rake gems:build. We can ask capistrano to run that command on the server by adding the following to your Capfile.


after "deploy:finalize_update" do
# build the native extensions for hpricot gem
run "cd #{release_path} && #{rake} RAILS_ENV=#{rails_env} gems:build GEM=hpricot"
end



Now when capistrano deploys in with all the other messages you'll see something like


...
* executing "cd /opt/apps/my_project/releases/20100108185109 && rake RAILS_ENV=production gems:build"
servers: ["your.server.com"]
[your.server.com] executing command
** [out :: your.server.com] (in /opt/apps/my_project/releases/20100108185109)
** [out :: your.server.com] Built gem: '/opt/apps/my_project/releases/20100108185109/vendor/gems/hpricot-0.8.2'
command finished
...



So rails give us a few simple patterns to follow to freeze our gems in the vendor folder and with a few lines in you Capfile you can use this pattern to vendor a gem with native extensions.

Freezing a gem that has native extensions

about 2 years ago | Alex Rothenberg: Common Sense Software

I like to freeze all the gems I use as we run in a shared hosting environment and need to our apps isolated from each other. Deployments are also handled by an operational team that does not intimately understand our applications so keeping our deployments to a single capistrano command cap deploy:migrations has been a big win for us. Freezing most gems is pretty straightforward and has been built in since Rails 2.1. When dealing with a gem that requires native extensions to be built there's only one additional step to add to your Capfil.

Let's say we want to localize hpricot which does include native C extensions.

First tell Rails about your gem by adding a config.gem line to your environment.rb


Rails::Initializer.run do |config|
...
config.gem 'hpricot'
...
end



Now we can ask rails about its configured gems


$ rake gems
(in /Users/alexrothenberg/ruby/my_project)
- [I] hpricot

I = Installed
F = Frozen
R = Framework (loaded before rails starts)



The 'I' means its hpricot is installed on my system but not frozen in the application. If you see '[]' instead you need to run sudo gem install hpricot (add '--source http://gemcutter.org' if necessary). At this point you could write some code to use hpricot and your application will work. But if hpricot (or the version you're expecting) is not installed on your production server you'll be in trouble.

To freeze the gem into your vendor directory run rake gems:unpack (optionally you can add 'GEM=hpricot' if you just want to unpack one gem).


$ rake gems:unpack
(in /Users/alexrothenberg/ruby/my_project)
Unpacked gem: '/Users/alexrothenberg/ruby/my_project/vendor/gems/hpricot-0.8.2'



We can ask rails again...


$ rake gems
(in /Users/alexrothenberg/ruby/my_project)
The following gems have native components that need to be built
hpricot

You're running:
ruby 1.8.6.287 at /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby
rubygems 1.3.2 at /Users/alexrothenberg/.gem/ruby/1.8, /Library/Ruby/Gems/1.8, /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/gems/1.8

Run `rake gems:build` to build the unbuilt gems.



Oops our vendored gem is missing hasn't built the native extensions. Not to worry the message tells us what to do and we run rake gems:build


$ rake gems:build
(in /Users/alexrothenberg/ruby/my_project)
Built gem: '/Users/alexrothenberg/ruby/mars-admin/vendor/gems/hpricot-0.8.2'
alex-rothenbergs:mars-admin alexrothenberg$ rake gems
(in /Users/alexrothenberg/ruby/my_project)
- [F] hpricot

I = Installed
F = Frozen
R = Framework (loaded before rails starts)



We can ask rails again to see that the gem is now frozen and also look in our vendor folder


$ rake gems
(in /Users/alexrothenberg/ruby/my_project)
- [F] hpricot

I = Installed
F = Frozen
R = Framework (loaded before rails starts)

$ ls vendor/gems/hpricot-0.8.2/
total 72
-rw-r--r-- 1 alexrothenberg staff 4672 Jan 13 12:33 CHANGELOG
-rw-r--r-- 1 alexrothenberg staff 1048 Jan 13 12:33 COPYING
-rw-r--r-- 1 alexrothenberg staff 9216 Jan 13 12:33 README
-rw-r--r-- 1 alexrothenberg staff 8242 Jan 13 12:33 Rakefile
drwxr-xr-x 4 alexrothenberg staff 136 Jan 13 12:33 ext/
drwxr-xr-x 3 alexrothenberg staff 102 Jan 13 12:33 extras/
drwxr-xr-x 6 alexrothenberg staff 204 Jan 13 12:39 lib/
drwxr-xr-x 11 alexrothenberg staff 374 Jan 13 12:33 test/



Everything looks good and you can check this into git and now have a frozen version of the hpricot gem stored with your application.
But if we stop here, when we deploy to our production server we'd be using the native extensions we built on your laptop which may not work on the server if you have one is 32bit and the other 64bit or you have different OS libraries installed or any number of other reasons.

To be safe, we need to rebuild the native extensions on the server when we deploy. This is not as hard as it sounds as rails gave us the rake task rake gems:build. We can ask capistrano to run that command on the server by adding the following to your Capfile.


after "deploy:finalize_update" do
# build the native extensions for hpricot gem
run "cd #{release_path} && #{rake} RAILS_ENV=#{rails_env} gems:build GEM=hpricot"
end



Now when capistrano deploys in with all the other messages you'll see something like


...
* executing "cd /opt/apps/my_project/releases/20100108185109 && rake RAILS_ENV=production gems:build"
servers: ["your.server.com"]
[your.server.com] executing command
** [out :: your.server.com] (in /opt/apps/my_project/releases/20100108185109)
** [out :: your.server.com] Built gem: '/opt/apps/my_project/releases/20100108185109/vendor/gems/hpricot-0.8.2'
command finished
...



So rails give us a few simple patterns to follow to freeze our gems in the vendor folder and with a few lines in you Capfile you can use this pattern to vendor a gem with native extensions.

Generate Models from Tables - Legacy Data Gem

over 2 years ago | Alex Rothenberg: Common Sense Software

Today I'd like to announce the release of a gem I've been working on Legacy Data

Getting started on a Rails project with a large existing database can be daunting. How to you extract all the information that's
encoded in the database? Do you have to understand the entire data model before you get started? The models_from_tables generator
in the legacy_data gem can help! This generator looks into your existing database and generates ActiveRecord models based on the
information encoded in it.

How to use it



  • To generate an ActiveRecord model for each table in the database just type
    script/generate models_from_tables

  • If you don't want all tables in the database tell it which table to model
    script/generate models_from_tables --table-name comments
    This uses any foreign_key constraints in the database to spider the database and model the comments table and all associated tables.

  • If you really only want the comments table tell it not to follow any foreign_keys
    script/generate models_from_tables --table-name comments --skip-associated

  • If you use factory girl (and everyone should) it will generate a simple factory for each model it generates
    script/generate models_from_tables --table-name comments --with-factories


(You do need to install the plugin gem install legacy_data as long as http://gemcutter.org is one of your gem sources)

Examples


Several examples come with the gem source in the examples folder on github. These include


  • A simple blog database tested with MySQL and Sqlite3

  • The Drupal 6.14 database tested with MySQL

  • The J2EE Petstore example tested with MySQL, Sqlite3 and Oracle


What kind of information can it extract from the database?


Associations


If the database contains foreign_key constraints it uses them to build has_many or belongs_to associations
in your ActiveRecord models

Validation constraints


It will generate the following types of validation constraints in your models


  • validates_uniqueness_of - For columns where the database has an index that enforces uniqueness

  • validates_presence_of - When the database column is non-nullable

  • validates_inclusion_of - For non-nullable boolean columns and custom constraints with a SQL rule flag IN ('Y', 'N')

  • validates_numericality_of - For integer columns (nullable and non-nullable)

  • custom validation - For custom SQL validation rules in the database it puts a placeholder in your model with the original SQL for you to translate into Ruby


Non-Rails naming conventions


Since the database is existing it's likely that it doesn't follow Rails naming conventions. Not to worry as the generator will
put the non-standard name into the generated models if it needs to.


What kinds of non-standard names can it generate?

Let's look at a sample output

class Post < ActiveRecord::Base

  set_table_name  :tbpost
  set_primary_key :postid
  
  # Relationships
  has_many :comments, :foreign_key => :postid

  # Constraints
  validates_presence_of :title, :body
  
end




  • Class Names - It named the model Post instead of the Rails convention Tbpost. The generator could not do this itself but knowing the conventions will often not apply to legacy databases it pauses after spidering the database giving you a chance to override the table to class name mapping. It generates a yaml file app/models/table_mappings.yml where you can verify or change any class name before proceeding to generate the models.

  • Table Names - It overrode the table name since the actual name tbpost does not match the Rails naming convention posts

  • Primary Keys - It overrode the primary key since the actual column postid does not match the Rails naming convention id

  • Foreign Keys - It overrode the foreign key on the comment table to be postid instead of the Rails naming convention id

Generate Models from Tables - Legacy Data Gem

over 2 years ago | Alex Rothenberg: Common Sense Software

Today I'd like to announce the release of a gem I've been working on Legacy Data

Getting started on a Rails project with a large existing database can be daunting. How to you extract all the information that's
encoded in the database? Do you have to understand the entire data model before you get started? The models_from_tables generator
in the legacy_data gem can help! This generator looks into your existing database and generates ActiveRecord models based on the
information encoded in it.

How to use it



  • To generate an ActiveRecord model for each table in the database just type
    script/generate models_from_tables

  • If you don't want all tables in the database tell it which table to model
    script/generate models_from_tables --table-name comments
    This uses any foreign_key constraints in the database to spider the database and model the comments table and all associated tables.

  • If you really only want the comments table tell it not to follow any foreign_keys
    script/generate models_from_tables --table-name comments --skip-associated

  • If you use factory girl (and everyone should) it will generate a simple factory for each model it generates
    script/generate models_from_tables --table-name comments --with-factories


(You do need to install the plugin gem install legacy_data as long as http://gemcutter.org is one of your gem sources)

Examples


Several examples come with the gem source in the examples folder on github. These include


  • A simple blog database tested with MySQL and Sqlite3

  • The Drupal 6.14 database tested with MySQL

  • The J2EE Petstore example tested with MySQL, Sqlite3 and Oracle


What kind of information can it extract from the database?


Associations


If the database contains foreign_key constraints it uses them to build has_many or belongs_to associations
in your ActiveRecord models

Validation constraints


It will generate the following types of validation constraints in your models


  • validates_uniqueness_of - For columns where the database has an index that enforces uniqueness

  • validates_presence_of - When the database column is non-nullable

  • validates_inclusion_of - For non-nullable boolean columns and custom constraints with a SQL rule flag IN ('Y', 'N')

  • validates_numericality_of - For integer columns (nullable and non-nullable)

  • custom validation - For custom SQL validation rules in the database it puts a placeholder in your model with the original SQL for you to translate into Ruby


Non-Rails naming conventions


Since the database is existing it's likely that it doesn't follow Rails naming conventions. Not to worry as the generator will
put the non-standard name into the generated models if it needs to.


What kinds of non-standard names can it generate?

Let's look at a sample output

class Post < ActiveRecord::Base

  set_table_name  :tbpost
  set_primary_key :postid
  
  # Relationships
  has_many :comments, :foreign_key => :postid

  # Constraints
  validates_presence_of :title, :body
  
end




  • Class Names - It named the model Post instead of the Rails convention Tbpost. The generator could not do this itself but knowing the conventions will often not apply to legacy databases it pauses after spidering the database giving you a chance to override the table to class name mapping. It generates a yaml file app/models/table_mappings.yml where you can verify or change any class name before proceeding to generate the models.

  • Table Names - It overrode the table name since the actual name tbpost does not match the Rails naming convention posts

  • Primary Keys - It overrode the primary key since the actual column postid does not match the Rails naming convention id

  • Foreign Keys - It overrode the foreign key on the comment table to be postid instead of the Rails naming convention id

PL/SQL debugging information now a part of Oracle Enhanced Adapter v1.2.2

over 2 years ago | Alex Rothenberg: Common Sense Software

Raimonds Simanovskis has just published a version 1.2.2 of the Oracle Enhanced Adapter that includes the ability to capture dbms_output debug statements from your pl/sql code in the Rails log file. This is a bit of code that I wrote and blogged about a few months ago so not only do I think its useful but am very excited to have contributed to something many others use.

PL/SQL debugging information now a part of Oracle Enhanced Adapter v1.2.2

over 2 years ago | Alex Rothenberg: Common Sense Software

Raimonds Simanovskis has just published a version 1.2.2 of the Oracle Enhanced Adapter that includes the ability to capture dbms_output debug statements from your pl/sql code in the Rails log file. This is a bit of code that I wrote and blogged about a few months ago so not only do I think its useful but am very excited to have contributed to something many others use.

Using the Whenever gem to manage scheduled cron jobs without installing it on the server

over 2 years ago | Alex Rothenberg: Common Sense Software

I've been using Javan's Whenever gem to manage scheduled jobs in my project and its fantastic!! There are many existing resources where you can learn more (readme, railscast google group) but I'd like to describe the specific way I'm using it


  • When the gem is not installed on my server

  • How administrators can use Capistrano to both schedule and unschedule your jobs

  • How to use a library such as the Oracle client that requires certain environment variables



At the end of the day we want to have 2 capistrano tasks we can run to have cron call a rake task of ours on the schedule we want.

  cap schedule_jobs
  cap unschedule_jobs



If we want to get fancy and pass a custom configuration argument that will be passed into the rake task.

  cap schedule_jobs SOME_CONFIGURATION=false



When the gem is not installed on my server


Let's start with the Capfile

#Capfile
desc "Schedule the jobs"
task :schedule_job, :roles => :app, :only => { :primary => true } do
  some_configuration = ENV['SOME_CONFIGURATION'] || true #default to true
  arguments = ["RAILS_ENV=#{rails_env}",
               "APP_PATH=#{current_path}",
               "SOME_CONFIGURATION=#{some_configuration}"].join(' ')
  run "cd #{current_path} && #{rake} whenever:update_crontab #{arguments}"
end

desc "Unschedule the jobs"
task :unschedule_job, :roles => :app, :only => { :primary => true } do
  run "cd #{current_path} && #{rake} whenever:update_crontab  UNSCHEDULE=true"
end



How does this differ from the example on the whenever site? Since the gem is not installed on the server we cannot call whenever from the command line so invoke the whenever:update_crontab rake task instead and to allow administrators to easily disable the scheduled jobs we define the unschedule_jobs capistrano task. Let's take a look at the whenever:update_crontab Rake task that gets this all done.

#lib/tasks/whenever.rake
namespace :whenever do
  desc "updates crontab with our scheduled jobs"  
  task :update_crontab => :load_whenever_gem do     
    Whenever::CommandLine.execute({:update=>true, :identifier=>'YOUR_APP_NAME'})
  end

  task :load_whenever_gem do     
    begin
      gem_dir_root = "#{RAILS_ROOT}/vendor/gems/"
      chronic_gem_dir = Dir["#{RAILS_ROOT}/vendor/gems/*"].detect do |subdir|
        subdir.gsub(gem_dir_root,"") =~ /^(\w+-)?chronic-(\d+)/ && File.exist?("#{subdir}/lib/chronic.rb")
      end
      require "#{chronic_gem_dir}/lib/chronic"

      whenever_gem_dir = Dir["#{RAILS_ROOT}/vendor/gems/*"].detect do |subdir|
        subdir.gsub(gem_dir_root,"") =~ /^(\w+-)?whenever-(\d+)/ && File.exist?("#{subdir}/lib/whenever.rb")
      end
      require "#{whenever_gem_dir}/lib/whenever"

    rescue MissingSourceFile => e
      raise "Cannot find Whenever or Chronic : #{e}"
    end
  end
end



The actual whenever:update_crontab task just does the same the command line does but unless you add config.gem 'whenever' to your environment (which I don't since whenever is not needed by my app at runtime) we also have the other task that loads whenever and chronic from the vendor/gems directory.

At this point we've gotten Capistrano calling a rake task to invoke whenever even though the gem is localized in my application but not installed on the server.

How administrators can use capistrano to both schedule and unschedule your jobs



The whenever gem does not have any support for unscheduling but it will schedule whatever is included in your schedule.rb file so if that file tells it to schedule nothing that's the same as unscheduling. Its easy to do that by wrapping the entire file with unless ENV['UNSCHEDULE'] (remember the UNSCHEDULE parameter we passed in the Capfile?)

#config/schedule.rb
unless ENV['UNSCHEDULE']

  module MyApp
    module Job
      class CronRakeTask < Whenever::Job::Default
        def output
          path_required
          "cd #{@path} && /usr/bin/env /usr/local/bin/cron_rake #{task} SOME_CONFIGURATION=#{ENV['SOME_CONFIGURATION']} RAILS_ENV=#{@environment}"
        end      
      end
    end
  end

  set :path,        ENV['APP_PATH'] || RAILS_ROOT 
  set :environment, RAILS_ENV || 'production'
  every 1.day, :at => '10:00pm' do
    command 'do_something', :class       => MyApp::Job::CronRakeTask, 
                            :environment => @environment, 
                            :path        => @path
  end
end



What's going on with the weird CronRakeTask? This brings us to the final point

How to use a library such as the Oracle client that requires certain environment variables



Cron loads its environment settings differently than an interactive shell and typically does not have all the environment variables you may have in your profile file. You could solve this by adding them to the top of your crontab file but I prefer to leave that file as simple as possible and create a wrapper script to call instead of rake. Basically the CronRakeTask does the same as Whenever's built in RakeTask except it calls /usr/local/bin/cron_rake instead of rake.

The cron_rake file just sets the environment variables I need then calls rake.

#!/bin/bash

export PATH=/usr/local/lib/ruby-enterprise/bin:$PATH
export ORACLE_HOME=/opt/oracle
export LD_LIBRARY_PATH=/opt/oracle:$LD_LIBRARY_PATH

rake $@



As I said in the beginning since I've been using the Whenever gem I no longer need to manually edit my crontab files ever and I can enable my jobs as part of my normal deployment process. Its wonderful and I think everyone should use it!

Using the Whenever gem to manage scheduled cron jobs without installing it on the server

over 2 years ago | Alex Rothenberg: Common Sense Software

I've been using Javan's Whenever gem to manage scheduled jobs in my project and its fantastic!! There are many existing resources where you can learn more (readme, railscast google group) but I'd like to describe the specific way I'm using it


  • When the gem is not installed on my server

  • How administrators can use Capistrano to both schedule and unschedule your jobs

  • How to use a library such as the Oracle client that requires certain environment variables



At the end of the day we want to have 2 capistrano tasks we can run to have cron call a rake task of ours on the schedule we want.

  cap schedule_jobs
  cap unschedule_jobs



If we want to get fancy and pass a custom configuration argument that will be passed into the rake task.

  cap schedule_jobs SOME_CONFIGURATION=false



When the gem is not installed on my server


Let's start with the Capfile

#Capfile
desc "Schedule the jobs"
task :schedule_job, :roles => :app, :only => { :primary => true } do
  some_configuration = ENV['SOME_CONFIGURATION'] || true #default to true
  arguments = ["RAILS_ENV=#{rails_env}",
               "APP_PATH=#{current_path}",
               "SOME_CONFIGURATION=#{some_configuration}"].join(' ')
  run "cd #{current_path} && #{rake} whenever:update_crontab #{arguments}"
end

desc "Unschedule the jobs"
task :unschedule_job, :roles => :app, :only => { :primary => true } do
  run "cd #{current_path} && #{rake} whenever:update_crontab  UNSCHEDULE=true"
end



How does this differ from the example on the whenever site? Since the gem is not installed on the server we cannot call whenever from the command line so invoke the whenever:update_crontab rake task instead and to allow administrators to easily disable the scheduled jobs we define the unschedule_jobs capistrano task. Let's take a look at the whenever:update_crontab Rake task that gets this all done.

#lib/tasks/whenever.rake
namespace :whenever do
  desc "updates crontab with our scheduled jobs"  
  task :update_crontab => :load_whenever_gem do     
    Whenever::CommandLine.execute({:update=>true, :identifier=>'YOUR_APP_NAME'})
  end

  task :load_whenever_gem do     
    begin
      gem_dir_root = "#{RAILS_ROOT}/vendor/gems/"
      chronic_gem_dir = Dir["#{RAILS_ROOT}/vendor/gems/*"].detect do |subdir|
        subdir.gsub(gem_dir_root,"") =~ /^(\w+-)?chronic-(\d+)/ && File.exist?("#{subdir}/lib/chronic.rb")
      end
      require "#{chronic_gem_dir}/lib/chronic"

      whenever_gem_dir = Dir["#{RAILS_ROOT}/vendor/gems/*"].detect do |subdir|
        subdir.gsub(gem_dir_root,"") =~ /^(\w+-)?whenever-(\d+)/ && File.exist?("#{subdir}/lib/whenever.rb")
      end
      require "#{whenever_gem_dir}/lib/whenever"

    rescue MissingSourceFile => e
      raise "Cannot find Whenever or Chronic : #{e}"
    end
  end
end



The actual whenever:update_crontab task just does the same the command line does but unless you add config.gem 'whenever' to your environment (which I don't since whenever is not needed by my app at runtime) we also have the other task that loads whenever and chronic from the vendor/gems directory.

At this point we've gotten Capistrano calling a rake task to invoke whenever even though the gem is localized in my application but not installed on the server.

How administrators can use capistrano to both schedule and unschedule your jobs



The whenever gem does not have any support for unscheduling but it will schedule whatever is included in your schedule.rb file so if that file tells it to schedule nothing that's the same as unscheduling. Its easy to do that by wrapping the entire file with unless ENV['UNSCHEDULE'] (remember the UNSCHEDULE parameter we passed in the Capfile?)

#config/schedule.rb
unless ENV['UNSCHEDULE']

  module MyApp
    module Job
      class CronRakeTask < Whenever::Job::Default
        def output
          path_required
          "cd #{@path} && /usr/bin/env /usr/local/bin/cron_rake #{task} SOME_CONFIGURATION=#{ENV['SOME_CONFIGURATION']} RAILS_ENV=#{@environment}"
        end      
      end
    end
  end

  set :path,        ENV['APP_PATH'] || RAILS_ROOT 
  set :environment, RAILS_ENV || 'production'
  every 1.day, :at => '10:00pm' do
    command 'do_something', :class       => MyApp::Job::CronRakeTask, 
                            :environment => @environment, 
                            :path        => @path
  end
end



What's going on with the weird CronRakeTask? This brings us to the final point

How to use a library such as the Oracle client that requires certain environment variables



Cron loads its environment settings differently than an interactive shell and typically does not have all the environment variables you may have in your profile file. You could solve this by adding them to the top of your crontab file but I prefer to leave that file as simple as possible and create a wrapper script to call instead of rake. Basically the CronRakeTask does the same as Whenever's built in RakeTask except it calls /usr/local/bin/cron_rake instead of rake.

The cron_rake file just sets the environment variables I need then calls rake.

#!/bin/bash

export PATH=/usr/local/lib/ruby-enterprise/bin:$PATH
export ORACLE_HOME=/opt/oracle
export LD_LIBRARY_PATH=/opt/oracle:$LD_LIBRARY_PATH

rake $@



As I said in the beginning since I've been using the Whenever gem I no longer need to manually edit my crontab files ever and I can enable my jobs as part of my normal deployment process. Its wonderful and I think everyone should use it!

How to capture Oracle's dbms_output in your Rails log file

over 2 years ago | Alex Rothenberg: Common Sense Software

I have been writing a rails application on top of a large existing Oracle database where each table has 5+ triggers that each call several stored procedures and each of those PL/SQL stored procedures is hundreds of lines long. Often a simple update statement fails with an ORA-xxxx exception coming from deep in the PL/SQL code and it can be tough to figure out what's gone wrong.

The usual way Oracle database folks figure out what's going on is to put print statements in their code. In oracle this looks like

  dbms_output.put_line('hi i hit this line of pl/sql');
When you're using an Oracle editor like TOAD or SQLDeveloper you have to turn output on and then will see anything that's printed.
  set serveroutput on;
This is great if you divide the application between Rails
and Database developers and assume each group can work independently to write perfect code but what about the real world!
Today I want to show you how I monkey patched the Oracle Enhanced Adapter to stick the dbms_output into
the rails log file.
Let's start with an simple example of a simple PL/SQL function that tells you if a string is more than 5 characters long (with some simple debugging print statements).
  CREATE OR REPLACE FUNCTION
MORE_THAN_FIVE_CHARACTERS_LONG (some_text VARCHAR2) RETURN INTEGER
AS
longer_than_five INTEGER;
BEGIN
dbms_output.put_line('before the if -' || some_text || '-');
IF length(some_text) > 5 THEN
dbms_output.put_line('it is longer than 5');
longer_than_five := 1;
ELSE
dbms_output.put_line('it is 5 or shorter');
longer_than_five := 0;
END IF;
dbms_output.put_line('about to return: ' || longer_than_five);
RETURN longer_than_five;
END;
Now we can run the following in TOAD
  set serveroutput on;
select MORE_THAN_FIVE_CHARACTERS_LONG('a long string') from dual;
select MORE_THAN_FIVE_CHARACTERS_LONG('short') from dual;
And we get this output
  MORE_THAN_FIVE_CHARACTERS_LONG('ALONGSTRING') 
---------------------------------------------
1

1 rows selected

before the if -a long string-
it is longer than 5
about to return: 1

MORE_THAN_FIVE_CHARACTERS_LONG('SHORT')
---------------------------------------
0

1 rows selected

before the if -short-
it is 5 or shorter
about to return: 0
To get the same in our Rails app we just need to monkey patch the OracleEnhancedAdapter by copying what's below into your project as config/initializers/oracle_enhanced_adapter.rb

  module ActiveRecord
module ConnectionAdapters
class OracleEnhancedAdapter < AbstractAdapter
DBMS_OUTPUT_BUFFER_SIZE = 10000 #can be 1-1000000
DBMS_LINE_MAX_SIZE = 1000
def enable_dbms_output
@enable_dbms_output = true
execute "BEGIN dbms_output.enable(#{DBMS_OUTPUT_BUFFER_SIZE}); END;"
end
def disable_dbms_output
@enable_dbms_output = false
execute "BEGIN dbms_output.disable(); END;"
end
def dbms_output_enabled?
@enable_dbms_output
end

protected
def log(sql, name)
super(sql, name)
ensure
log_all_dbms_output if dbms_output_enabled?
end

private
def log_next_line_of_dbms_output
dbms_output_text, status = @connection.exec "BEGIN dbms_output.get_line(:return, :status); END;", ' '*DBMS_LINE_MAX_SIZE, 1
got_text = (status == 0)
@logger.debug "DBMS_OUTPUT: #{dbms_output_text}" if got_text
got_text
end

def log_all_dbms_output
while log_next_line_of_dbms_output do
end
end
end
end
end

To use it let's show a quick script/console session
  >> ActiveRecord::Base.connection.enable_dbms_output
=> []
>> ActiveRecord::Base.connection.select_all("select MORE_THAN_FIVE_CHARACTERS_LONG('a long string') from dual")
=> [{"more_than_five_characters_long('alongstring')"=>1}]
And what's in log/development.log

SQL (27.0ms) BEGIN dbms_output.enable(10000); END;
SQL (25.9ms) select MORE_THAN_FIVE_CHARACTERS_LONG('a long string') from dual
DBMS_OUTPUT: before the if -a long string-
DBMS_OUTPUT: it is longer than 5
DBMS_OUTPUT: about to return: 1

This is admittedly a very simple example but I have used this in a real application where I am updating several related ActiveRecord objects and seeing the DBMS_OUTPUT inline with the various SQL update statements has been extremely useful in tracking down a real bug in the PL/SQL procedure that has been in our system for over a year!

I've submitted this as a patch to the Oracle Enhanced Adapter so perhaps it will be included at some point so you wont have to do the monkey patching yourself. I was somewhat surprised not to find anything similar out there so if you know of something please leave a comment.

How to capture Oracle's dbms_output in your Rails log file

over 2 years ago | Alex Rothenberg: Common Sense Software

I have been writing a rails application on top of a large existing Oracle database where each table has 5+ triggers that each call several stored procedures and each of those PL/SQL stored procedures is hundreds of lines long. Often a simple update statement fails with an ORA-xxxx exception coming from deep in the PL/SQL code and it can be tough to figure out what's gone wrong.

The usual way Oracle database folks figure out what's going on is to put print statements in their code. In oracle this looks like

  dbms_output.put_line('hi i hit this line of pl/sql');
When you're using an Oracle editor like TOAD or SQLDeveloper you have to turn output on and then will see anything that's printed.
  set serveroutput on;
This is great if you divide the application between Rails
and Database developers and assume each group can work independently to write perfect code but what about the real world!
Today I want to show you how I monkey patched the Oracle Enhanced Adapter to stick the dbms_output into
the rails log file.
Let's start with an simple example of a simple PL/SQL function that tells you if a string is more than 5 characters long (with some simple debugging print statements).
  CREATE OR REPLACE FUNCTION
MORE_THAN_FIVE_CHARACTERS_LONG (some_text VARCHAR2) RETURN INTEGER
AS
longer_than_five INTEGER;
BEGIN
dbms_output.put_line('before the if -' || some_text || '-');
IF length(some_text) > 5 THEN
dbms_output.put_line('it is longer than 5');
longer_than_five := 1;
ELSE
dbms_output.put_line('it is 5 or shorter');
longer_than_five := 0;
END IF;
dbms_output.put_line('about to return: ' || longer_than_five);
RETURN longer_than_five;
END;
Now we can run the following in TOAD
  set serveroutput on;
select MORE_THAN_FIVE_CHARACTERS_LONG('a long string') from dual;
select MORE_THAN_FIVE_CHARACTERS_LONG('short') from dual;
And we get this output
  MORE_THAN_FIVE_CHARACTERS_LONG('ALONGSTRING') 
---------------------------------------------
1

1 rows selected

before the if -a long string-
it is longer than 5
about to return: 1

MORE_THAN_FIVE_CHARACTERS_LONG('SHORT')
---------------------------------------
0

1 rows selected

before the if -short-
it is 5 or shorter
about to return: 0
To get the same in our Rails app we just need to monkey patch the OracleEnhancedAdapter by copying what's below into your project as config/initializers/oracle_enhanced_adapter.rb

  module ActiveRecord
module ConnectionAdapters
class OracleEnhancedAdapter < AbstractAdapter
DBMS_OUTPUT_BUFFER_SIZE = 10000 #can be 1-1000000
DBMS_LINE_MAX_SIZE = 1000
def enable_dbms_output
@enable_dbms_output = true
execute "BEGIN dbms_output.enable(#{DBMS_OUTPUT_BUFFER_SIZE}); END;"
end
def disable_dbms_output
@enable_dbms_output = false
execute "BEGIN dbms_output.disable(); END;"
end
def dbms_output_enabled?
@enable_dbms_output
end

protected
def log(sql, name)
super(sql, name)
ensure
log_all_dbms_output if dbms_output_enabled?
end

private
def log_next_line_of_dbms_output
dbms_output_text, status = @connection.exec "BEGIN dbms_output.get_line(:return, :status); END;", ' '*DBMS_LINE_MAX_SIZE, 1
got_text = (status == 0)
@logger.debug "DBMS_OUTPUT: #{dbms_output_text}" if got_text
got_text
end

def log_all_dbms_output
while log_next_line_of_dbms_output do
end
end
end
end
end

To use it let's show a quick script/console session
  >> ActiveRecord::Base.connection.enable_dbms_output
=> []
>> ActiveRecord::Base.connection.select_all("select MORE_THAN_FIVE_CHARACTERS_LONG('a long string') from dual")
=> [{"more_than_five_characters_long('alongstring')"=>1}]
And what's in log/development.log

SQL (27.0ms) BEGIN dbms_output.enable(10000); END;
SQL (25.9ms) select MORE_THAN_FIVE_CHARACTERS_LONG('a long string') from dual
DBMS_OUTPUT: before the if -a long string-
DBMS_OUTPUT: it is longer than 5
DBMS_OUTPUT: about to return: 1

This is admittedly a very simple example but I have used this in a real application where I am updating several related ActiveRecord objects and seeing the DBMS_OUTPUT inline with the various SQL update statements has been extremely useful in tracking down a real bug in the PL/SQL procedure that has been in our system for over a year!

I've submitted this as a patch to the Oracle Enhanced Adapter so perhaps it will be included at some point so you wont have to do the monkey patching yourself. I was somewhat surprised not to find anything similar out there so if you know of something please leave a comment.

One year of blogging

over 2 years ago | Alex Rothenberg: Common Sense Software

I just noticed that its been a year since my first post and thought I'd take a moment to reflect on the experience....

I set out with a goal of writing 2-3 posts a month and have managed to write 28 articles over 12 months so I feel pretty good about that. Most of the time I've gotten excited about something I accomplished or learned during the week. The process of writing it up turned out to be as much of a learning experience for myself as the original discovery. As they say "Teaching is the best way to learn"!

I've been most surprised how easy it is to get to the top of a Google Search results list (for very specific searches of course). With my Google Analytics tracking I can see more than 1/2 of my hits coming from search engines and not all of them are me searching for my own name :) Its been great when I've gotten comments from people I don't know (especially when the comments are positive).

Writing this blog has also helped turn my focus outward so I'm not only working on internal projects at my company but feel (in a small way) a part of the broader community.

I hope to continue in the coming year as I'm having a lot of fun so far ...

One year of blogging

over 2 years ago | Alex Rothenberg: Common Sense Software

I just noticed that its been a year since my first post and thought I'd take a moment to reflect on the experience....

I set out with a goal of writing 2-3 posts a month and have managed to write 28 articles over 12 months so I feel pretty good about that. Most of the time I've gotten excited about something I accomplished or learned during the week. The process of writing it up turned out to be as much of a learning experience for myself as the original discovery. As they say "Teaching is the best way to learn"!

I've been most surprised how easy it is to get to the top of a Google Search results list (for very specific searches of course). With my Google Analytics tracking I can see more than 1/2 of my hits coming from search engines and not all of them are me searching for my own name :) Its been great when I've gotten comments from people I don't know (especially when the comments are positive).

Writing this blog has also helped turn my focus outward so I'm not only working on internal projects at my company but feel (in a small way) a part of the broader community.

I hope to continue in the coming year as I'm having a lot of fun so far ...

Using Github Through Draconian Proxies (Windows And Unix)

over 2 years ago | Alex Rothenberg: Common Sense Software

I came across this great set of instructions on how to tunnel through a proxy to use github - http://returnbooleantrue.blogspot.com/2009/06/using-github-through-draconian-proxies.html

The proxy where I work is not quite so draconian that I need to follow these steps but its nice to have these instructions just in case :)

Using Github Through Draconian Proxies (Windows And Unix)

over 2 years ago | Alex Rothenberg: Common Sense Software

I came across this great set of instructions on how to tunnel through a proxy to use github - http://returnbooleantrue.blogspot.com/2009/06/using-github-through-draconian-proxies.html

The proxy where I work is not quite so draconian that I need to follow these steps but its nice to have these instructions just in case :)

Metric_fu 1.1.0 released with a patch from me

over 2 years ago | Alex Rothenberg: Common Sense Software

Metric_fu just released version 1.1.0 of their gem which I'm pleased to say includes a patch submitted by me. This is exciting as its the first time I've had my code included by someone I don't know in one of the open source projects I admire.

I was not originally going to make the patch as I thought I'd just hack around to fix the problem locally just enough to get it working but my friend Matt encouraged me to fix the root cause which turned out to be not too hard and got me into the metric_fu source. I always learn something when I read source from others I admire.

The process of submitting the patch was pretty easy and I plan to make it a habit when I run into issues with other gems. I,

  1. Experienced a problem and decided not to live with it
  2. Created a fork of metric_fu on github
  3. Cloned locally and iteratively made my changes until I had my fix complete
  4. Rebased my changes into a single commit (I wrote an article on this a few weeks ago)
  5. Pushed my patch back to github on a branch
  6. Submitted my patch back to Jake's repository as an issue

Solving the problem of gem dependencies with github names

I am using relevance-rcov instead of the rubyforge version of rcov as it has been maintained more recently and specifically fixed a segmentation fault problem I previously faced. The problem is that I couldn't install metric_fu as it put a gem dependency on 'rcov' in its gemspec. What seems broken is that there's no way to put a dependency on 'rcov' but have rubygems realize that 'relevance-rcov' is a fork of rcov so should satisfy the dependency.

I've seen a lot of talk recently about rubygems (including Why Using require ‘rubygems’ Is Wrong, RubyGems: Problems and (proposed) Solutions and Rip: A Next Generation Ruby Packaging System - Watch Out RubyGems!) but so far I haven't found a solution to the forking and naming problem emerge.

For Metric_fu I decided to change the install-time dependency to a runtime one. This solves my basic problem of not finding relevance-rcov but also allows for more flexibile use of metric_fu. If someone does not want to generate reek or rcov metrics why should they be forced to install those gems in order to use metric_fu? By defering the dependency until runtime we will never hit the dependency for those metrics we are not using.

You can look through the commit to see exactly how I did this.
  1. Deleted the install time dependencies from metric_fu.gemspec
  2. Added the runtime dependencies when a specific type of metric is instantiated by
    • Inserting a verify_dependencies! strategy step to the initialization in lib/base/generator.rb
    • Implementing verify_dependencies! in each of the subclasses for different metrics in lib/generators/*
  3. Did it with TDD writing tests around everything before implementing my changes

Should gems use install time dependencies? Should a gem author decide which fork of a gem is required? What do you think?

Metric_fu 1.1.0 released with a patch from me

over 2 years ago | Alex Rothenberg: Common Sense Software

Metric_fu just released version 1.1.0 of their gem which I'm pleased to say includes a patch submitted by me. This is exciting as its the first time I've had my code included by someone I don't know in one of the open source projects I admire.

I was not originally going to make the patch as I thought I'd just hack around to fix the problem locally just enough to get it working but my friend Matt encouraged me to fix the root cause which turned out to be not too hard and got me into the metric_fu source. I always learn something when I read source from others I admire.

The process of submitting the patch was pretty easy and I plan to make it a habit when I run into issues with other gems. I,

  1. Experienced a problem and decided not to live with it
  2. Created a fork of metric_fu on github
  3. Cloned locally and iteratively made my changes until I had my fix complete
  4. Rebased my changes into a single commit (I wrote an article on this a few weeks ago)
  5. Pushed my patch back to github on a branch
  6. Submitted my patch back to Jake's repository as an issue

Solving the problem of gem dependencies with github names

I am using relevance-rcov instead of the rubyforge version of rcov as it has been maintained more recently and specifically fixed a segmentation fault problem I previously faced. The problem is that I couldn't install metric_fu as it put a gem dependency on 'rcov' in its gemspec. What seems broken is that there's no way to put a dependency on 'rcov' but have rubygems realize that 'relevance-rcov' is a fork of rcov so should satisfy the dependency.

I've seen a lot of talk recently about rubygems (including Why Using require ‘rubygems’ Is Wrong, RubyGems: Problems and (proposed) Solutions and Rip: A Next Generation Ruby Packaging System - Watch Out RubyGems!) but so far I haven't found a solution to the forking and naming problem emerge.

For Metric_fu I decided to change the install-time dependency to a runtime one. This solves my basic problem of not finding relevance-rcov but also allows for more flexibile use of metric_fu. If someone does not want to generate reek or rcov metrics why should they be forced to install those gems in order to use metric_fu? By defering the dependency until runtime we will never hit the dependency for those metrics we are not using.

You can look through the commit to see exactly how I did this.
  1. Deleted the install time dependencies from metric_fu.gemspec
  2. Added the runtime dependencies when a specific type of metric is instantiated by
    • Inserting a verify_dependencies! strategy step to the initialization in lib/base/generator.rb
    • Implementing verify_dependencies! in each of the subclasses for different metrics in lib/generators/*
  3. Did it with TDD writing tests around everything before implementing my changes

Should gems use install time dependencies? Should a gem author decide which fork of a gem is required? What do you think?

Changing history with git rebase: How to combine several commits into one

over 2 years ago | Alex Rothenberg: Common Sense Software

When I look at commits on github I'm always impressed at how concise they are. When I read a commit I can understand the intent of the change without getting distracted by the author's journey to get there. In contrast when I look at my commits they tend to be smaller and more incremental and meandering as I work my way down some false starts until I get to the solution I want. I'm guessing that I'm not alone in the way I work and recently discovered git rebase and an helpful tutorial showing how I can continue to work in my meandering style but package my changes to hide the journey before publishing to the world on github.

Let me show you what I mean with some changes I recently made to metric_fu. Over the course of a few days I made 6 small commits as you can see below.


> git log

commit d4b18b16e982ac57741f7a0a12cb085bb9b0e840
Author: Alex Rothenberg
Date: Mon Jun 1 09:55:37 2009 -0400

reverted rakefile

commit ebda1cb67a2f0f0a85e51469d70919fa7c27d318
Author: Alex Rothenberg
Date: Mon Jun 1 09:49:05 2009 -0400

refactoring of verify_dependencies

commit 3d45903b64a772fd09ec07bf69880bdb29ae4944
Author: Alex Rothenberg
Date: Fri May 29 20:40:49 2009 -0400

runtime dependency check for all gems

commit 36e269bf8f0edccfb39cc767182406cdaa16a559
Author: Alex Rothenberg
Date: Fri May 29 20:24:16 2009 -0400

logic for checking dependencies in generator base

commit bb2ee9a983c6adf54a8c95ee5851c6f8d3bffaba
Author: Alex Rothenberg
Date: Fri May 29 19:38:21 2009 -0400

made rcov dependency figure itself out when generating rcov metrics

commit 750b000e5563e917f21eb1b7837e8001fb53f688
Author: Alex Rothenberg
Date: Fri May 29 16:42:12 2009 -0400

removed gem dependency on rcov - to allow use of relevance-rcov or other github forks

commit d6af5089adce9eeed4916a155c3bdaeb4be6771a
Author: Randy Souza
Date: Sat May 16 08:47:37 2009 +0800

Added a simple fix for cases where Saikuro results with nested information
cause metrics:all to crash

Signed-off-by: Jake Scruggs

...


I am going to combine all of these into a single commit using the incredible power of git rebase. I find the last commit I do not want to change (the one made by Randy Souza on May 16th) and issue a git rebase command with that id.


> git rebase --interactive d6af5089adce9eeed4916a155c3bdaeb4be6771a


Now my editor comes up showing the 6 changes since then giving me options of what to do. Its very powerful, I can reorder commits, combine commits, eliminate commits. I'm going back in time to change the past!


pick 750b000 removed gem dependency on rcov - to allow use of relevance-rcov or other github forks
pick bb2ee9a made rcov dependency figure itself out when generating rcov metrics
pick 36e269b logic for checking dependencies in generator base
pick 3d45903 runtime dependency check for all gems
pick ebda1cb refactoring of verify_dependencies
pick d4b18b1 reverted rakefile

# Rebase d6af508..d4b18b1 onto d6af508
#
# Commands:
# p, pick = use commit
# e, edit = use commit, but stop for amending
# s, squash = use commit, but meld into previous commit
#
# If you remove a line here THAT COMMIT WILL BE LOST.
# However, if you remove everything, the rebase will be aborted.
#


In my case I want to combine these into a single commit so I change all but the first pick command to a squash and save.


pick 750b000 removed gem dependency on rcov - to allow use of relevance-rcov or other github forks
squash bb2ee9a made rcov dependency figure itself out when generating rcov metrics
squash 36e269b logic for checking dependencies in generator base
squash 3d45903 runtime dependency check for all gems
squash ebda1cb refactoring of verify_dependencies
squash d4b18b1 reverted rakefile

# Rebase d6af508..d4b18b1 onto d6af508
#
# Commands:
# p, pick = use commit
# e, edit = use commit, but stop for amending
# s, squash = use commit, but meld into previous commit
#
# If you remove a line here THAT COMMIT WILL BE LOST.
# However, if you remove everything, the rebase will be aborted.
#


It now gives me a chance to edit the new commit message which it defaults to the original messages concatenated together. This really is a single commit so if one of the original commits changed a file and a subsequent one undid the change that file will no longer appear in the list of modified files.



# This is a combination of 6 commits.
# The first commit's message is:
removed gem dependency on rcov - to allow use of relevance-rcov or other github forks

# This is the 2nd commit message:

made rcov dependency figure itself out when generating rcov metrics

# This is the 3rd commit message:

logic for checking dependencies in generator base

# This is the 4th commit message:

runtime dependency check for all gems

# This is the 5th commit message:

refactoring of verify_dependencies

# This is the 6th commit message:

reverted rakefile

# Please enter the commit message for your changes. Lines starting
# with '#' will be ignored, and an empty message aborts the commit.
# Explicit paths specified without -i nor -o; assuming --only paths...
# Not currently on any branch.
# Changes to be committed:
# (use "git reset HEAD ..." to unstage)
#
# modified: Rakefile
# modified: lib/base/generator.rb
# modified: lib/generators/flay.rb
# modified: lib/generators/flog.rb
# modified: lib/generators/rcov.rb
# modified: lib/generators/reek.rb
# modified: lib/generators/roodi.rb
# modified: lib/generators/saikuro.rb
# modified: metric_fu.gemspec
# modified: spec/base/generator_spec.rb
# modified: spec/generators/flay_spec.rb
# modified: spec/generators/flog_spec.rb
# modified: spec/generators/reek_spec.rb
#



Let's edit this commit message to read something like


Changed gem dependencies from install-time in gemspec to runtime when each
of the generators is loaded. This allows use of github gems (i.e.
relevance-rcov instead of rcov) and also allows you to install only the
gems for the metrics you plan on using.


# Please enter the commit message for your changes. Lines starting
# with '#' will be ignored, and an empty message aborts the commit.
# Explicit paths specified without -i nor -o; assuming --only paths...
# Not currently on any branch.
# Changes to be committed:
# (use "git reset HEAD ..." to unstage)
#
# modified: Rakefile
# modified: lib/base/generator.rb
# modified: lib/generators/flay.rb
# modified: lib/generators/flog.rb
# modified: lib/generators/rcov.rb
# modified: lib/generators/reek.rb
# modified: lib/generators/roodi.rb
# modified: lib/generators/saikuro.rb
# modified: metric_fu.gemspec
# modified: spec/base/generator_spec.rb
# modified: spec/generators/flay_spec.rb
# modified: spec/generators/flog_spec.rb
# modified: spec/generators/reek_spec.rb
#


And that's it! Now when we check our history we see.


> git log

commit 6f389364eb972871867d3a71677b8eb7046541a2
Author: Alex Rothenberg
Date: Fri May 29 16:42:12 2009 -0400

Changed gem dependencies from install-time in gemspec to runtime when each
of the generators is loaded. This allows use of github gems (i.e.
relevance-rcov instead of rcov) and also allows you to install only the
gems for the metrics you plan on using.

commit d6af5089adce9eeed4916a155c3bdaeb4be6771a
Author: Randy Souza
Date: Sat May 16 08:47:37 2009 +0800

Added a simple fix for cases where Saikuro results with nested information
cause metrics:all to crash

Signed-off-by: Jake Scruggs

...


And this commit is how I pushed this patch to github for all the world to see.

Changing history with git rebase: How to combine several commits into one

over 2 years ago | Alex Rothenberg: Common Sense Software

When I look at commits on github I'm always impressed at how concise they are. When I read a commit I can understand the intent of the change without getting distracted by the author's journey to get there. In contrast when I look at my commits they tend to be smaller and more incremental and meandering as I work my way down some false starts until I get to the solution I want. I'm guessing that I'm not alone in the way I work and recently discovered git rebase and an helpful tutorial showing how I can continue to work in my meandering style but package my changes to hide the journey before publishing to the world on github.

Let me show you what I mean with some changes I recently made to metric_fu. Over the course of a few days I made 6 small commits as you can see below.


> git log

commit d4b18b16e982ac57741f7a0a12cb085bb9b0e840
Author: Alex Rothenberg
Date: Mon Jun 1 09:55:37 2009 -0400

reverted rakefile

commit ebda1cb67a2f0f0a85e51469d70919fa7c27d318
Author: Alex Rothenberg
Date: Mon Jun 1 09:49:05 2009 -0400

refactoring of verify_dependencies

commit 3d45903b64a772fd09ec07bf69880bdb29ae4944
Author: Alex Rothenberg
Date: Fri May 29 20:40:49 2009 -0400

runtime dependency check for all gems

commit 36e269bf8f0edccfb39cc767182406cdaa16a559
Author: Alex Rothenberg
Date: Fri May 29 20:24:16 2009 -0400

logic for checking dependencies in generator base

commit bb2ee9a983c6adf54a8c95ee5851c6f8d3bffaba
Author: Alex Rothenberg
Date: Fri May 29 19:38:21 2009 -0400

made rcov dependency figure itself out when generating rcov metrics

commit 750b000e5563e917f21eb1b7837e8001fb53f688
Author: Alex Rothenberg
Date: Fri May 29 16:42:12 2009 -0400

removed gem dependency on rcov - to allow use of relevance-rcov or other github forks

commit d6af5089adce9eeed4916a155c3bdaeb4be6771a
Author: Randy Souza
Date: Sat May 16 08:47:37 2009 +0800

Added a simple fix for cases where Saikuro results with nested information
cause metrics:all to crash

Signed-off-by: Jake Scruggs

...


I am going to combine all of these into a single commit using the incredible power of git rebase. I find the last commit I do not want to change (the one made by Randy Souza on May 16th) and issue a git rebase command with that id.


> git rebase --interactive d6af5089adce9eeed4916a155c3bdaeb4be6771a


Now my editor comes up showing the 6 changes since then giving me options of what to do. Its very powerful, I can reorder commits, combine commits, eliminate commits. I'm going back in time to change the past!


pick 750b000 removed gem dependency on rcov - to allow use of relevance-rcov or other github forks
pick bb2ee9a made rcov dependency figure itself out when generating rcov metrics
pick 36e269b logic for checking dependencies in generator base
pick 3d45903 runtime dependency check for all gems
pick ebda1cb refactoring of verify_dependencies
pick d4b18b1 reverted rakefile

# Rebase d6af508..d4b18b1 onto d6af508
#
# Commands:
# p, pick = use commit
# e, edit = use commit, but stop for amending
# s, squash = use commit, but meld into previous commit
#
# If you remove a line here THAT COMMIT WILL BE LOST.
# However, if you remove everything, the rebase will be aborted.
#


In my case I want to combine these into a single commit so I change all but the first pick command to a squash and save.


pick 750b000 removed gem dependency on rcov - to allow use of relevance-rcov or other github forks
squash bb2ee9a made rcov dependency figure itself out when generating rcov metrics
squash 36e269b logic for checking dependencies in generator base
squash 3d45903 runtime dependency check for all gems
squash ebda1cb refactoring of verify_dependencies
squash d4b18b1 reverted rakefile

# Rebase d6af508..d4b18b1 onto d6af508
#
# Commands:
# p, pick = use commit
# e, edit = use commit, but stop for amending
# s, squash = use commit, but meld into previous commit
#
# If you remove a line here THAT COMMIT WILL BE LOST.
# However, if you remove everything, the rebase will be aborted.
#


It now gives me a chance to edit the new commit message which it defaults to the original messages concatenated together. This really is a single commit so if one of the original commits changed a file and a subsequent one undid the change that file will no longer appear in the list of modified files.



# This is a combination of 6 commits.
# The first commit's message is:
removed gem dependency on rcov - to allow use of relevance-rcov or other github forks

# This is the 2nd commit message:

made rcov dependency figure itself out when generating rcov metrics

# This is the 3rd commit message:

logic for checking dependencies in generator base

# This is the 4th commit message:

runtime dependency check for all gems

# This is the 5th commit message:

refactoring of verify_dependencies

# This is the 6th commit message:

reverted rakefile

# Please enter the commit message for your changes. Lines starting
# with '#' will be ignored, and an empty message aborts the commit.
# Explicit paths specified without -i nor -o; assuming --only paths...
# Not currently on any branch.
# Changes to be committed:
# (use "git reset HEAD ..." to unstage)
#
# modified: Rakefile
# modified: lib/base/generator.rb
# modified: lib/generators/flay.rb
# modified: lib/generators/flog.rb
# modified: lib/generators/rcov.rb
# modified: lib/generators/reek.rb
# modified: lib/generators/roodi.rb
# modified: lib/generators/saikuro.rb
# modified: metric_fu.gemspec
# modified: spec/base/generator_spec.rb
# modified: spec/generators/flay_spec.rb
# modified: spec/generators/flog_spec.rb
# modified: spec/generators/reek_spec.rb
#



Let's edit this commit message to read something like


Changed gem dependencies from install-time in gemspec to runtime when each
of the generators is loaded. This allows use of github gems (i.e.
relevance-rcov instead of rcov) and also allows you to install only the
gems for the metrics you plan on using.


# Please enter the commit message for your changes. Lines starting
# with '#' will be ignored, and an empty message aborts the commit.
# Explicit paths specified without -i nor -o; assuming --only paths...
# Not currently on any branch.
# Changes to be committed:
# (use "git reset HEAD ..." to unstage)
#
# modified: Rakefile
# modified: lib/base/generator.rb
# modified: lib/generators/flay.rb
# modified: lib/generators/flog.rb
# modified: lib/generators/rcov.rb
# modified: lib/generators/reek.rb
# modified: lib/generators/roodi.rb
# modified: lib/generators/saikuro.rb
# modified: metric_fu.gemspec
# modified: spec/base/generator_spec.rb
# modified: spec/generators/flay_spec.rb
# modified: spec/generators/flog_spec.rb
# modified: spec/generators/reek_spec.rb
#


And that's it! Now when we check our history we see.


> git log

commit 6f389364eb972871867d3a71677b8eb7046541a2
Author: Alex Rothenberg
Date: Fri May 29 16:42:12 2009 -0400

Changed gem dependencies from install-time in gemspec to runtime when each
of the generators is loaded. This allows use of github gems (i.e.
relevance-rcov instead of rcov) and also allows you to install only the
gems for the metrics you plan on using.

commit d6af5089adce9eeed4916a155c3bdaeb4be6771a
Author: Randy Souza
Date: Sat May 16 08:47:37 2009 +0800

Added a simple fix for cases where Saikuro results with nested information
cause metrics:all to crash

Signed-off-by: Jake Scruggs

...


And this commit is how I pushed this patch to github for all the world to see.

How to use dates in Rails when your database stores a string

almost 3 years ago | Alex Rothenberg: Common Sense Software

When working with Rails there's a lot of magic that happens behind the scenes to make it easy to do complex things. Most of the time you don't need to know how that "magic" works but there are times when things don't work as expected and its helpful to dig in and understand what Rails is doing under the covers so you can change how it works. Did I just say "change how Rails works"?!? I did! Rails is opinionated software that seeks to lead you down the golden path but there are legitimate times when you have to veer off that path and Rails lets you do so. I find this most often happens to me when I'm dealing with an existing legacy database which is not setup as Rails would like.

Today I'm going to go through an example that happened to me recently when I had an existing database that stored some dates in a text column but I needed to treat them as dates in my UI. I couldn't change the type of that column as there was another legacy application that expected it to be text.

Using Dates the Rails Way

First let's look at how easy it is to work with dates when you can follow the Rails Way. Let's create a new project and add a scaffolded Person object with a date attribute called birthday.

rails date_select_example
cd date_select_example
script/generate scaffold person name:string birthday:date
rake db:migrate



Now if we hit the site and try to create a new person we see a screen like this



And when you click "Create"



Exactly what you want with almost no code in the view and none in the model. I did make a minor edit to the view so we would get 1960 in the year select by adding :start_year=>1900 (a full documentation of date_select options are available here)

#app/views/people/new.html.erb
<h1>New person
<% form_for(@person) do |f| %>
  <%= f.error_messages %>
  <p>
    <%= f.label :name %><br />
    <%= f.text_field :name %>
  </p>
  <p>
    <%= f.label :birthday %><br />
    <%= f.date_select :birthday, :start_year=>1900 %>
  </p>
  <p>
    <%= f.submit 'Create' %>
  </p>
<% end %>

<%= link_to 'Back', people_path %>



and

#app/models/person.rb
class Person < ActiveRecord::Base
end



Errors when storing as text in the database

Now what happens when you run into a case where the date is stored as a string in the database. Let's say we have an "anniversary" attribute stored as a string that we want to treat the same way as we did birthday. We create the migration.

script/generate migration AddAnniversaryToPerson anniversary:string
rake db:migrate



Then add the anniversary to our view.

#app/views/people/edit.html.erb
<h1>Editing person
<% form_for(@person) do |f| %>
  <%= f.error_messages %>
  <p>
    <%= f.label :name %><br />
    <%= f.text_field :name %>
  </p>
  <p>
    <%= f.label :birthday %><br />
    <%= f.date_select :birthday, :start_year=>1900  %>
  </p>
  <p>
    <%= f.label :anniversary %><br />
    <%= f.date_select :anniversary, :start_year=>1900  %>
  </p>
  <p>
    <%= f.submit 'Update' %>
  </p>
<% end %>

<%= link_to 'Show', @person %> |
<%= link_to 'Back', people_path %>





It looks like we're done so we click "Update" and .. Oops. It doesn't work! We get the error 1 error(s) on assignment of multiparameter attributes. Now we need to figure out what multiparameter attributes are and whey they're not working for us.

ActiveRecord::MultiparameterAssignmentErrors in PeopleController#update

1 error(s) on assignment of multiparameter attributes
RAILS_ROOT: /Users/alexrothenberg/date_select_example

Application Trace | Framework Trace | Full Trace
/Library/Ruby/Gems/1.8/gems/activerecord-2.3.2/lib/active_record/base.rb:3061:in `execute_callstack_for_multiparameter_attributes'
/Library/Ruby/Gems/1.8/gems/activerecord-2.3.2/lib/active_record/base.rb:3022:in `assign_multiparameter_attributes'
/Library/Ruby/Gems/1.8/gems/activerecord-2.3.2/lib/active_record/base.rb:2749:in `attributes='
/Library/Ruby/Gems/1.8/gems/activerecord-2.3.2/lib/active_record/base.rb:2627:in `update_attributes'
/Users/alexrothenberg/date_select_example/app/controllers/people_controller.rb:63:in `update'
...more stack trace...

Request Parameters:

{"commit"=>"Update",
 "_method"=>"put",
 "authenticity_token"=>"qezkVq+MNzFuXxFBJ/GaSoh2BNdxM6oF3H7JP5beFFE=",
 "id"=>"1",
 "person"=>{"name"=>"Barack Obama",
 "birthday(2i)"=>"8",
 "birthday(3i)"=>"4",
 "anniversary(1i)"=>"2009",
 "anniversary(2i)"=>"5",
 "anniversary(3i)"=>"22",
 "birthday(1i)"=>"1960"}}



What happened? There are two keys to figuring out what's going on


  1. The date_select helper actually sends 3 http parameters to our application anniversary(1i), anniversary(2i) and anniversary(3i). ActiveRecord must combine those into a single Date before updating the row in the database.
  2. Looking at assign_multiparameter_attributes in active_record/base.rb we see this this comment that talks about combining 3 http parameters into a date type by calling new on the column type
    Instantiates objects for all attribute classes that needs more than one constructor parameter. This is done by calling new on the column type or aggregation type (through composed_of) object with these parameters.
    So having the pairs written_on(1) = "2004", written_on(2) = "6", written_on(3) = "24", will instantiate written_on (a date type) with Date.new("2004", "6", "24"). You can also specify a typecast character in the parentheses to have the parameters typecasted before they're used in the constructor. Use i for Fixnum, f for Float, s for String, and a for Array. If all the values for a given attribute are empty, the attribute will be set to nil.



Now we know we're close to the right place and can check the column's return type using script/console

  $ script/console
  Loading development environment (Rails 2.3.2)
  >> Person.columns_hash['birthday']
  => #
  >> Person.columns_hash['birthday'].klass
  => Date
  >> Person.columns_hash['anniversary']
  => #
  >> Person.columns_hash['anniversary'].klass
  => String



This makes sense. ActiveRecord is treating the anniversary column as a string because that's what it is in the database so is not combining the 3 multiparameter attributes into a Date. What we need to do is override the default ActiveRecord logic and tell it to treat this column as a date. We can write a failing spec

require File.dirname(__FILE__) + '/../spec_helper'

describe Person do
  it "should treat anniversary as a Date column" do
    Person.columns_hash['anniversary'].klass.should == Date
  end
end

# 'Person should treat anniversary as a Date column' FAILED
# expected: Date,
#      got: String (using ==)



Then take advantage of a trick of Ruby that allows you to extend an object without affecting other instances of its class (see PickAxe book's explanation of this technique). If we lookup the definition of klass for a column It is actually very simple to implement our fix with the 5 lines below.

class Person < ActiveRecord::Base
  class << columns_hash['anniversary']
    def type
      :date
    end
  end
end



Now our tests pass and when we go back to our site and click the Update button and it works.




How to use dates in Rails when your database stores a string

almost 3 years ago | Alex Rothenberg: Common Sense Software

When working with Rails there's a lot of magic that happens behind the scenes to make it easy to do complex things. Most of the time you don't need to know how that "magic" works but there are times when things don't work as expected and its helpful to dig in and understand what Rails is doing under the covers so you can change how it works. Did I just say "change how Rails works"?!? I did! Rails is opinionated software that seeks to lead you down the golden path but there are legitimate times when you have to veer off that path and Rails lets you do so. I find this most often happens to me when I'm dealing with an existing legacy database which is not setup as Rails would like.

Today I'm going to go through an example that happened to me recently when I had an existing database that stored some dates in a text column but I needed to treat them as dates in my UI. I couldn't change the type of that column as there was another legacy application that expected it to be text.

Using Dates the Rails Way

First let's look at how easy it is to work with dates when you can follow the Rails Way. Let's create a new project and add a scaffolded Person object with a date attribute called birthday.

rails date_select_example
cd date_select_example
script/generate scaffold person name:string birthday:date
rake db:migrate



Now if we hit the site and try to create a new person we see a screen like this



And when you click "Create"



Exactly what you want with almost no code in the view and none in the model. I did make a minor edit to the view so we would get 1960 in the year select by adding :start_year=>1900 (a full documentation of date_select options are available here)

#app/views/people/new.html.erb
<h1>New person
<% form_for(@person) do |f| %>
  <%= f.error_messages %>
  <p>
    <%= f.label :name %><br />
    <%= f.text_field :name %>
  </p>
  <p>
    <%= f.label :birthday %><br />
    <%= f.date_select :birthday, :start_year=>1900 %>
  </p>
  <p>
    <%= f.submit 'Create' %>
  </p>
<% end %>

<%= link_to 'Back', people_path %>



and

#app/models/person.rb
class Person < ActiveRecord::Base
end



Errors when storing as text in the database

Now what happens when you run into a case where the date is stored as a string in the database. Let's say we have an "anniversary" attribute stored as a string that we want to treat the same way as we did birthday. We create the migration.

script/generate migration AddAnniversaryToPerson anniversary:string
rake db:migrate



Then add the anniversary to our view.

#app/views/people/edit.html.erb
<h1>Editing person
<% form_for(@person) do |f| %>
  <%= f.error_messages %>
  <p>
    <%= f.label :name %><br />
    <%= f.text_field :name %>
  </p>
  <p>
    <%= f.label :birthday %><br />
    <%= f.date_select :birthday, :start_year=>1900  %>
  </p>
  <p>
    <%= f.label :anniversary %><br />
    <%= f.date_select :anniversary, :start_year=>1900  %>
  </p>
  <p>
    <%= f.submit 'Update' %>
  </p>
<% end %>

<%= link_to 'Show', @person %> |
<%= link_to 'Back', people_path %>





It looks like we're done so we click "Update" and .. Oops. It doesn't work! We get the error 1 error(s) on assignment of multiparameter attributes. Now we need to figure out what multiparameter attributes are and whey they're not working for us.

ActiveRecord::MultiparameterAssignmentErrors in PeopleController#update

1 error(s) on assignment of multiparameter attributes
RAILS_ROOT: /Users/alexrothenberg/date_select_example

Application Trace | Framework Trace | Full Trace
/Library/Ruby/Gems/1.8/gems/activerecord-2.3.2/lib/active_record/base.rb:3061:in `execute_callstack_for_multiparameter_attributes'
/Library/Ruby/Gems/1.8/gems/activerecord-2.3.2/lib/active_record/base.rb:3022:in `assign_multiparameter_attributes'
/Library/Ruby/Gems/1.8/gems/activerecord-2.3.2/lib/active_record/base.rb:2749:in `attributes='
/Library/Ruby/Gems/1.8/gems/activerecord-2.3.2/lib/active_record/base.rb:2627:in `update_attributes'
/Users/alexrothenberg/date_select_example/app/controllers/people_controller.rb:63:in `update'
...more stack trace...

Request Parameters:

{"commit"=>"Update",
 "_method"=>"put",
 "authenticity_token"=>"qezkVq+MNzFuXxFBJ/GaSoh2BNdxM6oF3H7JP5beFFE=",
 "id"=>"1",
 "person"=>{"name"=>"Barack Obama",
 "birthday(2i)"=>"8",
 "birthday(3i)"=>"4",
 "anniversary(1i)"=>"2009",
 "anniversary(2i)"=>"5",
 "anniversary(3i)"=>"22",
 "birthday(1i)"=>"1960"}}



What happened? There are two keys to figuring out what's going on


  1. The date_select helper actually sends 3 http parameters to our application anniversary(1i), anniversary(2i) and anniversary(3i). ActiveRecord must combine those into a single Date before updating the row in the database.
  2. Looking at assign_multiparameter_attributes in active_record/base.rb we see this this comment that talks about combining 3 http parameters into a date type by calling new on the column type
    Instantiates objects for all attribute classes that needs more than one constructor parameter. This is done by calling new on the column type or aggregation type (through composed_of) object with these parameters.
    So having the pairs written_on(1) = "2004", written_on(2) = "6", written_on(3) = "24", will instantiate written_on (a date type) with Date.new("2004", "6", "24"). You can also specify a typecast character in the parentheses to have the parameters typecasted before they're used in the constructor. Use i for Fixnum, f for Float, s for String, and a for Array. If all the values for a given attribute are empty, the attribute will be set to nil.



Now we know we're close to the right place and can check the column's return type using script/console

  $ script/console
  Loading development environment (Rails 2.3.2)
  >> Person.columns_hash['birthday']
  => #
  >> Person.columns_hash['birthday'].klass
  => Date
  >> Person.columns_hash['anniversary']
  => #
  >> Person.columns_hash['anniversary'].klass
  => String



This makes sense. ActiveRecord is treating the anniversary column as a string because that's what it is in the database so is not combining the 3 multiparameter attributes into a Date. What we need to do is override the default ActiveRecord logic and tell it to treat this column as a date. We can write a failing spec

require File.dirname(__FILE__) + '/../spec_helper'

describe Person do
  it "should treat anniversary as a Date column" do
    Person.columns_hash['anniversary'].klass.should == Date
  end
end

# 'Person should treat anniversary as a Date column' FAILED
# expected: Date,
#      got: String (using ==)



Then take advantage of a trick of Ruby that allows you to extend an object without affecting other instances of its class (see PickAxe book's explanation of this technique). If we lookup the definition of klass for a column It is actually very simple to implement our fix with the 5 lines below.

class Person < ActiveRecord::Base
  class << columns_hash['anniversary']
    def type
      :date
    end
  end
end



Now our tests pass and when we go back to our site and click the Update button and it works.




Testing AJAX without a browser with Cucumber and Webrat

almost 3 years ago | Alex Rothenberg: Common Sense Software

I have lately fallen in love with using Cucumber and Webrat for my integration/acceptance testing. Cucumber because it allows non-technical people to write or at least read the test scenarios and Webrat because it matches content and encourages you to write integration tests without relying on xpath to find html elements. The way I like to use these tools is to run Rails integration tests which means its fast since I don’t need to start a mongrel or fire up a browser and can use Rails’ transactional fixtures to rollback all my database changes at the end of each test scenario. The only downside is that you can’t test javascript.

Today I am going to talk about how to get around this and test a form with an ajax autocomplete field. I've built a sample application with all the code examples here and you can download it from http://github.com/alexrothenberg/testing-ajax-example if you like. The application I'm building is just some simple app created with scaffolding that just has a User resource with a name and address. I modified the /users page to not display all users but include the auto_complete typeahead to let you pick a user (imagining there may be a lot) so the page looks something like this.



My first test scenario will ignore the ajax and just test the form which is super easy and can be done by writing a single feature file.

#features/find_a_user.feature
Feature: Allow anyone to find a user and see their details
  In order to handle a large set of users
  I want search with autocomplete
 
  Scenario: View a candidate detail page without testing ajax
    Given "Mickey Mouse" is a user living at "123 Main Street"
    When I am on the homepage
      And I fill in "Which user" with "Mickey Mouse"
      And I press "Find"
    Then I should see "Mickey Mouse"
      And I should see "123 Main Street"



I run it it all passes and I get

$ rake features
(in /Users/alexrothenberg/ruby/testing-ajax-example)
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby -I "/Library/Ruby/Gems/1.8/gems/cucumber-0.3.2/lib:lib" 
"/Library/Ruby/Gems/1.8/gems/cucumber-0.3.2/bin/cucumber" --format pretty --require features/step_definitions/user_steps.rb 
--require features/step_definitions/webrat_steps.rb --require features/support/env.rb --require features/support/paths.rb 
features/find_a_user.feature
Feature: Allow anyone to find a user and see their details
  In order to handle a large set of users 
  I want search with autocomplete

  Scenario: View a candidate detail page without testing ajax     # features/find_a_user.feature:5
    Given "Mickey Mouse" is a user living at "123 Main Street"    # features/step_definitions/user_steps.rb:1
    When I am on the homepage                                     # features/step_definitions/webrat_steps.rb:6
    And I fill in "Which user" with "Mickey Mouse"                # features/step_definitions/webrat_steps.rb:22
    And I press "Find"                                            # features/step_definitions/webrat_steps.rb:14
    Then I should see "Mickey Mouse"                              # features/step_definitions/webrat_steps.rb:93
    And I should see "123 Main Street"                            # features/step_definitions/webrat_steps.rb:93

1 scenario (1 passed)
6 steps (6 passed)



I makes use of the default webrat steps that cucumber gives you for free in features/steps/webrat_steps.rb so I don’t even have to write any code to get it to pass but I still haven’t tested any of my code that responds to the autocomplete request. I’d like my test to verify that my routes, controller and model will all work together. So, I write another scenario and run it and this time it fails because I haven't defined the typeahead steps for the typeahead lines.

#first scenario omitted

Scenario: View a candidate detail page testing (most of) the ajax # features/find_a_user.feature:13
    Given "Mickey Mouse" is a user living at "123 Main Street"      # features/step_definitions/user_steps.rb:1
    When I am on the homepage                                       # features/step_definitions/webrat_steps.rb:6
    And I typeahead in "Which user" with "Mickey Mouse"             # features/find_a_user.feature:16
    And I fill in "Which candidate" with the first typeahead result # features/find_a_user.feature:17
    And I press "Find"                                              # features/step_definitions/webrat_steps.rb:14
    Then I should see "Mickey Mouse"                                # features/step_definitions/webrat_steps.rb:93
    And I should see "123 Main Street"                              # features/step_definitions/webrat_steps.rb:93

2 scenarios (1 undefined, 1 passed)
13 steps (3 skipped, 2 undefined, 8 passed)

You can implement step definitions for undefined steps with these snippets:

When /^I typeahead in "([^\"]*)" with "([^\"]*)"$/ do |arg1, arg2|
  pending
end

When /^I fill in "([^\"]*)" with the first typeahead result$/ do |arg1|
  pending
end



Now I take the hints cucumber has given me and write my autocomplete steps. The interesting thing here is that I need to leave the response object unchanged so I can fill in the form field after running the typeahead step so I can't use the existing webrat steps as they work on a single pair of request and response objects. So I knew I'd be creating a new class with its own request and response that could be used without affecting the one used by my other cucumber steps. Using good outside-in development practices I deferred thinking about how to do that and first wrote my steps file to look something like this. One interesting thing to notice here is that you can invoke a step from inside another step just by omitting the block as I do in the second step.

When /^I typeahead in "(.*)" with "(.*)"$/ do |field, value|
  field = field_labeled field
  @typeahead = AutoCompleteStepHelper.new(request)
  @typeahead.type(field, value)
end

When /^I fill in "(.*)" with the first typeahead result$/ do |field|
  When %Q[I fill in "#{field}" with "#{@typeahead.items.first}"]
end



Now when run the feature again it fails telling me I haven’t yet built the AutoCompleteStepHelper.

#first scenario omitted

Scenario: View a candidate detail page testing (most of) the ajax # features/find_a_user.feature:13
    Given "Mickey Mouse" is a user living at "123 Main Street"      # features/step_definitions/user_steps.rb:1
    When I am on the homepage                                       # features/step_definitions/webrat_steps.rb:6
    And I typeahead in "Which user" with "Mickey Mouse"             # features/step_definitions/autocomplete_steps.rb:1
      uninitialized constant AutoCompleteStepHelper (NameError)
      ./features/step_definitions/autocomplete_steps.rb:3:in `/^I typeahead in "(.*)" with "(.*)"$/'
      features/find_a_user.feature:16:in `And I typeahead in "Which user" with "Mickey Mouse"'
    And I fill in "Which candidate" with the first typeahead result # features/step_definitions/autocomplete_steps.rb:7
    And I press "Find"                                              # features/step_definitions/webrat_steps.rb:14
    Then I should see "Mickey Mouse"                                # features/step_definitions/webrat_steps.rb:93
    And I should see "123 Main Street"                              # features/step_definitions/webrat_steps.rb:93

2 scenarios (1 failed, 1 passed)
13 steps (1 failed, 4 skipped, 8 passed)



So the next step is to write the AutoCompleteStepHelper class. This class’ job is to have its own request and response that will not affect the ones used by cucumber for the main page requests. It turns out that I can do this by having my class extend ActionController::IntegrationTest and I can even use webrat methods in it because webrat adds its methods to IntegrationTest. In this example I'm calling visit and current_dom and using nokogiri to parse the dom. It is a little weird that I'm subclassing IntegrationTest but this class is not a TestUnit class itself but I decided that was okay.

class AutoCompleteStepsHelper < ActionController::IntegrationTest
  def initialize(existing_request)
    @controller_name = existing_request.parameters[:controller]
    @controller_class = "#{@controller_name.to_s.camelize}Controller".constantize
    raise "Can't determine controller class for #{@controller_class_name}" if @controller_class.nil?

    @controller = @controller_class.new
    @request = ActionController::TestRequest.new
    @response = ActionController::TestResponse.new
    @response.session = @request.session
  end
 
  def type(field, value)
    visit url_for(:controller=>@controller_name, :action=>"auto_complete_for_#{field.id}", field.send(:name)=>value)
  end
 
  def items
    current_dom.search('//ul/li').map(&:inner_html)
  end
end



Now when I run the feature it all passes.

#first scenario omitted

  Scenario: View a candidate detail page testing (most of) the ajax # features/find_a_user.feature:13
    Given "Mickey Mouse" is a user living at "123 Main Street"      # features/step_definitions/user_steps.rb:1
    When I am on the homepage                                       # features/step_definitions/webrat_steps.rb:6
    And I typeahead in "Which user" with "Mickey Mouse"             # features/step_definitions/autocomplete_steps.rb:1
    And I fill in "Which user" with the first typeahead result      # features/step_definitions/autocomplete_steps.rb:7
    And I press "Find"                                              # features/step_definitions/webrat_steps.rb:14
    Then I should see "Mickey Mouse"                                # features/step_definitions/webrat_steps.rb:93
    And I should see "123 Main Street"                              # features/step_definitions/webrat_steps.rb:93

2 scenarios (2 passed)
13 steps (13 passed)



The last step I took was to test that the list returned in the typeahead was correct. I created another scenario in my feature.

Scenario: Typeahead should return 2 users that match but not a third
    Given "Mickey Mouse" is a user living at "123 Main Street"
      And "Donald Duck" is a user living at "123 Pond Lane"
      And "Minnie Mouse" is a user living at "123 Disney Avenue"
    When I am on the homepage
      And I typeahead in "Which user" with "Mi"
    Then I should see in my typeahead "Mickey Mouse"
      And I should see in my typeahead "Minnie Mouse"
      And I should not see in my typeahead "Donald Duck"



and I added two new steps

Then /^I should see in my typeahead "(.*)"$/ do |text|
  @typeahead.response_body.should =~ /#{text}/m
end

Then /^I should not see in my typeahead "(.*)"$/ do |text|
  @typeahead.response_body.should_not =~ /#{text}/m
end



Now when I run my features all 3 scenarios are passing

$ rake features
(in /Users/alexrothenberg/ruby/testing-ajax-example)
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby -I "/Library/Ruby/Gems/1.8/gems/cucumber-0.3.2/lib:lib" 
"/Library/Ruby/Gems/1.8/gems/cucumber-0.3.2/bin/cucumber" --format pretty --require features/step_definitions/autocomplete_steps.rb 
--require features/step_definitions/user_steps.rb --require features/step_definitions/webrat_steps.rb 
--require features/support/autocomplete_steps_helper.rb --require features/support/env.rb --require features/support/paths.rb 
features/find_a_user.feature

Feature: Allow anyone to find a user and see their details
  In order to handle a large set of users 
  I want search with autocomplete

  Scenario: View a candidate detail page without testing ajax  # features/find_a_user.feature:5
    Given "Mickey Mouse" is a user living at "123 Main Street" # features/step_definitions/user_steps.rb:1
    When I am on the homepage                                  # features/step_definitions/webrat_steps.rb:6
    And I fill in "Which user" with "Mickey Mouse"             # features/step_definitions/webrat_steps.rb:22
    And I press "Find"                                         # features/step_definitions/webrat_steps.rb:14
    Then I should see "Mickey Mouse"                           # features/step_definitions/webrat_steps.rb:93
    And I should see "123 Main Street"                         # features/step_definitions/webrat_steps.rb:93

  Scenario: View a candidate detail page testing (most of) the ajax # features/find_a_user.feature:13
    Given "Mickey Mouse" is a user living at "123 Main Street"      # features/step_definitions/user_steps.rb:1
    When I am on the homepage                                       # features/step_definitions/webrat_steps.rb:6
    And I typeahead in "Which user" with "Mickey Mouse"             # features/step_definitions/autocomplete_steps.rb:1
    And I fill in "Which user" with the first typeahead result      # features/step_definitions/autocomplete_steps.rb:7
    And I press "Find"                                              # features/step_definitions/webrat_steps.rb:14
    Then I should see "Mickey Mouse"                                # features/step_definitions/webrat_steps.rb:93
    And I should see "123 Main Street"                              # features/step_definitions/webrat_steps.rb:93

  Scenario: Typeahead should return 2 users that match but not a third # features/find_a_user.feature:22
    Given "Mickey Mouse" is a user living at "123 Main Street"         # features/step_definitions/user_steps.rb:1
    And "Donald Duck" is a user living at "123 Pond Lane"              # features/step_definitions/user_steps.rb:1
    And "Minnie Mouse" is a user living at "123 Disney Avenue"         # features/step_definitions/user_steps.rb:1
    When I am on the homepage                                          # features/step_definitions/webrat_steps.rb:6
    And I typeahead in "Which user" with "Mi"                          # features/step_definitions/autocomplete_steps.rb:1
    Then I should see in my typeahead "Mickey Mouse"                   # features/step_definitions/autocomplete_steps.rb:16
    And I should see in my typeahead "Minnie Mouse"                    # features/step_definitions/autocomplete_steps.rb:16
    And I should not see in my typeahead "Donald Duck"                 # features/step_definitions/autocomplete_steps.rb:20

3 scenarios (3 passed)
21 steps (21 passed)



What I've done is not full javascript testing (for that I'm planning to look into Blue-Ridge from Relevance). This technique does allow you to test ajax (skipping the "J") without a browser.

Testing AJAX without a browser with Cucumber and Webrat

almost 3 years ago | Alex Rothenberg: Common Sense Software

I have lately fallen in love with using Cucumber and Webrat for my integration/acceptance testing. Cucumber because it allows non-technical people to write or at least read the test scenarios and Webrat because it matches content and encourages you to write integration tests without relying on xpath to find html elements. The way I like to use these tools is to run Rails integration tests which means its fast since I don’t need to start a mongrel or fire up a browser and can use Rails’ transactional fixtures to rollback all my database changes at the end of each test scenario. The only downside is that you can’t test javascript.

Today I am going to talk about how to get around this and test a form with an ajax autocomplete field. I've built a sample application with all the code examples here and you can download it from http://github.com/alexrothenberg/testing-ajax-example if you like. The application I'm building is just some simple app created with scaffolding that just has a User resource with a name and address. I modified the /users page to not display all users but include the auto_complete typeahead to let you pick a user (imagining there may be a lot) so the page looks something like this.



My first test scenario will ignore the ajax and just test the form which is super easy and can be done by writing a single feature file.

#features/find_a_user.feature
Feature: Allow anyone to find a user and see their details
  In order to handle a large set of users
  I want search with autocomplete
 
  Scenario: View a candidate detail page without testing ajax
    Given "Mickey Mouse" is a user living at "123 Main Street"
    When I am on the homepage
      And I fill in "Which user" with "Mickey Mouse"
      And I press "Find"
    Then I should see "Mickey Mouse"
      And I should see "123 Main Street"



I run it it all passes and I get

$ rake features
(in /Users/alexrothenberg/ruby/testing-ajax-example)
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby -I "/Library/Ruby/Gems/1.8/gems/cucumber-0.3.2/lib:lib" 
"/Library/Ruby/Gems/1.8/gems/cucumber-0.3.2/bin/cucumber" --format pretty --require features/step_definitions/user_steps.rb 
--require features/step_definitions/webrat_steps.rb --require features/support/env.rb --require features/support/paths.rb 
features/find_a_user.feature
Feature: Allow anyone to find a user and see their details
  In order to handle a large set of users 
  I want search with autocomplete

  Scenario: View a candidate detail page without testing ajax     # features/find_a_user.feature:5
    Given "Mickey Mouse" is a user living at "123 Main Street"    # features/step_definitions/user_steps.rb:1
    When I am on the homepage                                     # features/step_definitions/webrat_steps.rb:6
    And I fill in "Which user" with "Mickey Mouse"                # features/step_definitions/webrat_steps.rb:22
    And I press "Find"                                            # features/step_definitions/webrat_steps.rb:14
    Then I should see "Mickey Mouse"                              # features/step_definitions/webrat_steps.rb:93
    And I should see "123 Main Street"                            # features/step_definitions/webrat_steps.rb:93

1 scenario (1 passed)
6 steps (6 passed)



I makes use of the default webrat steps that cucumber gives you for free in features/steps/webrat_steps.rb so I don’t even have to write any code to get it to pass but I still haven’t tested any of my code that responds to the autocomplete request. I’d like my test to verify that my routes, controller and model will all work together. So, I write another scenario and run it and this time it fails because I haven't defined the typeahead steps for the typeahead lines.

#first scenario omitted

Scenario: View a candidate detail page testing (most of) the ajax # features/find_a_user.feature:13
    Given "Mickey Mouse" is a user living at "123 Main Street"      # features/step_definitions/user_steps.rb:1
    When I am on the homepage                                       # features/step_definitions/webrat_steps.rb:6
    And I typeahead in "Which user" with "Mickey Mouse"             # features/find_a_user.feature:16
    And I fill in "Which candidate" with the first typeahead result # features/find_a_user.feature:17
    And I press "Find"                                              # features/step_definitions/webrat_steps.rb:14
    Then I should see "Mickey Mouse"                                # features/step_definitions/webrat_steps.rb:93
    And I should see "123 Main Street"                              # features/step_definitions/webrat_steps.rb:93

2 scenarios (1 undefined, 1 passed)
13 steps (3 skipped, 2 undefined, 8 passed)

You can implement step definitions for undefined steps with these snippets:

When /^I typeahead in "([^\"]*)" with "([^\"]*)"$/ do |arg1, arg2|
  pending
end

When /^I fill in "([^\"]*)" with the first typeahead result$/ do |arg1|
  pending
end



Now I take the hints cucumber has given me and write my autocomplete steps. The interesting thing here is that I need to leave the response object unchanged so I can fill in the form field after running the typeahead step so I can't use the existing webrat steps as they work on a single pair of request and response objects. So I knew I'd be creating a new class with its own request and response that could be used without affecting the one used by my other cucumber steps. Using good outside-in development practices I deferred thinking about how to do that and first wrote my steps file to look something like this. One interesting thing to notice here is that you can invoke a step from inside another step just by omitting the block as I do in the second step.

When /^I typeahead in "(.*)" with "(.*)"$/ do |field, value|
  field = field_labeled field
  @typeahead = AutoCompleteStepHelper.new(request)
  @typeahead.type(field, value)
end

When /^I fill in "(.*)" with the first typeahead result$/ do |field|
  When %Q[I fill in "#{field}" with "#{@typeahead.items.first}"]
end



Now when run the feature again it fails telling me I haven’t yet built the AutoCompleteStepHelper.

#first scenario omitted

Scenario: View a candidate detail page testing (most of) the ajax # features/find_a_user.feature:13
    Given "Mickey Mouse" is a user living at "123 Main Street"      # features/step_definitions/user_steps.rb:1
    When I am on the homepage                                       # features/step_definitions/webrat_steps.rb:6
    And I typeahead in "Which user" with "Mickey Mouse"             # features/step_definitions/autocomplete_steps.rb:1
      uninitialized constant AutoCompleteStepHelper (NameError)
      ./features/step_definitions/autocomplete_steps.rb:3:in `/^I typeahead in "(.*)" with "(.*)"$/'
      features/find_a_user.feature:16:in `And I typeahead in "Which user" with "Mickey Mouse"'
    And I fill in "Which candidate" with the first typeahead result # features/step_definitions/autocomplete_steps.rb:7
    And I press "Find"                                              # features/step_definitions/webrat_steps.rb:14
    Then I should see "Mickey Mouse"                                # features/step_definitions/webrat_steps.rb:93
    And I should see "123 Main Street"                              # features/step_definitions/webrat_steps.rb:93

2 scenarios (1 failed, 1 passed)
13 steps (1 failed, 4 skipped, 8 passed)



So the next step is to write the AutoCompleteStepHelper class. This class’ job is to have its own request and response that will not affect the ones used by cucumber for the main page requests. It turns out that I can do this by having my class extend ActionController::IntegrationTest and I can even use webrat methods in it because webrat adds its methods to IntegrationTest. In this example I'm calling visit and current_dom and using nokogiri to parse the dom. It is a little weird that I'm subclassing IntegrationTest but this class is not a TestUnit class itself but I decided that was okay.

class AutoCompleteStepsHelper < ActionController::IntegrationTest
  def initialize(existing_request)
    @controller_name = existing_request.parameters[:controller]
    @controller_class = "#{@controller_name.to_s.camelize}Controller".constantize
    raise "Can't determine controller class for #{@controller_class_name}" if @controller_class.nil?

    @controller = @controller_class.new
    @request = ActionController::TestRequest.new
    @response = ActionController::TestResponse.new
    @response.session = @request.session
  end
 
  def type(field, value)
    visit url_for(:controller=>@controller_name, :action=>"auto_complete_for_#{field.id}", field.send(:name)=>value)
  end
 
  def items
    current_dom.search('//ul/li').map(&:inner_html)
  end
end



Now when I run the feature it all passes.

#first scenario omitted

  Scenario: View a candidate detail page testing (most of) the ajax # features/find_a_user.feature:13
    Given "Mickey Mouse" is a user living at "123 Main Street"      # features/step_definitions/user_steps.rb:1
    When I am on the homepage                                       # features/step_definitions/webrat_steps.rb:6
    And I typeahead in "Which user" with "Mickey Mouse"             # features/step_definitions/autocomplete_steps.rb:1
    And I fill in "Which user" with the first typeahead result      # features/step_definitions/autocomplete_steps.rb:7
    And I press "Find"                                              # features/step_definitions/webrat_steps.rb:14
    Then I should see "Mickey Mouse"                                # features/step_definitions/webrat_steps.rb:93
    And I should see "123 Main Street"                              # features/step_definitions/webrat_steps.rb:93

2 scenarios (2 passed)
13 steps (13 passed)



The last step I took was to test that the list returned in the typeahead was correct. I created another scenario in my feature.

Scenario: Typeahead should return 2 users that match but not a third
    Given "Mickey Mouse" is a user living at "123 Main Street"
      And "Donald Duck" is a user living at "123 Pond Lane"
      And "Minnie Mouse" is a user living at "123 Disney Avenue"
    When I am on the homepage
      And I typeahead in "Which user" with "Mi"
    Then I should see in my typeahead "Mickey Mouse"
      And I should see in my typeahead "Minnie Mouse"
      And I should not see in my typeahead "Donald Duck"



and I added two new steps

Then /^I should see in my typeahead "(.*)"$/ do |text|
  @typeahead.response_body.should =~ /#{text}/m
end

Then /^I should not see in my typeahead "(.*)"$/ do |text|
  @typeahead.response_body.should_not =~ /#{text}/m
end



Now when I run my features all 3 scenarios are passing

$ rake features
(in /Users/alexrothenberg/ruby/testing-ajax-example)
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby -I "/Library/Ruby/Gems/1.8/gems/cucumber-0.3.2/lib:lib" 
"/Library/Ruby/Gems/1.8/gems/cucumber-0.3.2/bin/cucumber" --format pretty --require features/step_definitions/autocomplete_steps.rb 
--require features/step_definitions/user_steps.rb --require features/step_definitions/webrat_steps.rb 
--require features/support/autocomplete_steps_helper.rb --require features/support/env.rb --require features/support/paths.rb 
features/find_a_user.feature

Feature: Allow anyone to find a user and see their details
  In order to handle a large set of users 
  I want search with autocomplete

  Scenario: View a candidate detail page without testing ajax  # features/find_a_user.feature:5
    Given "Mickey Mouse" is a user living at "123 Main Street" # features/step_definitions/user_steps.rb:1
    When I am on the homepage                                  # features/step_definitions/webrat_steps.rb:6
    And I fill in "Which user" with "Mickey Mouse"             # features/step_definitions/webrat_steps.rb:22
    And I press "Find"                                         # features/step_definitions/webrat_steps.rb:14
    Then I should see "Mickey Mouse"                           # features/step_definitions/webrat_steps.rb:93
    And I should see "123 Main Street"                         # features/step_definitions/webrat_steps.rb:93

  Scenario: View a candidate detail page testing (most of) the ajax # features/find_a_user.feature:13
    Given "Mickey Mouse" is a user living at "123 Main Street"      # features/step_definitions/user_steps.rb:1
    When I am on the homepage                                       # features/step_definitions/webrat_steps.rb:6
    And I typeahead in "Which user" with "Mickey Mouse"             # features/step_definitions/autocomplete_steps.rb:1
    And I fill in "Which user" with the first typeahead result      # features/step_definitions/autocomplete_steps.rb:7
    And I press "Find"                                              # features/step_definitions/webrat_steps.rb:14
    Then I should see "Mickey Mouse"                                # features/step_definitions/webrat_steps.rb:93
    And I should see "123 Main Street"                              # features/step_definitions/webrat_steps.rb:93

  Scenario: Typeahead should return 2 users that match but not a third # features/find_a_user.feature:22
    Given "Mickey Mouse" is a user living at "123 Main Street"         # features/step_definitions/user_steps.rb:1
    And "Donald Duck" is a user living at "123 Pond Lane"              # features/step_definitions/user_steps.rb:1
    And "Minnie Mouse" is a user living at "123 Disney Avenue"         # features/step_definitions/user_steps.rb:1
    When I am on the homepage                                          # features/step_definitions/webrat_steps.rb:6
    And I typeahead in "Which user" with "Mi"                          # features/step_definitions/autocomplete_steps.rb:1
    Then I should see in my typeahead "Mickey Mouse"                   # features/step_definitions/autocomplete_steps.rb:16
    And I should see in my typeahead "Minnie Mouse"                    # features/step_definitions/autocomplete_steps.rb:16
    And I should not see in my typeahead "Donald Duck"                 # features/step_definitions/autocomplete_steps.rb:20

3 scenarios (3 passed)
21 steps (21 passed)



What I've done is not full javascript testing (for that I'm planning to look into Blue-Ridge from Relevance). This technique does allow you to test ajax (skipping the "J") without a browser.

Managing the draw of a single-elimination tournament

almost 3 years ago | Alex Rothenberg: Common Sense Software

My company has been running a GO tournament and since its around NCAA March Madness time I thought it'd be easy to find a site to manage the draw. I looked but couldn't find anything that did what I wanted so I wrote my own. Caveat: I just spent a few days on this and it could stand to be enhanced with more features and a rewrite of the ui implementation but I'm open to suggestions if anyone finds this useful.

To use it


  1. Edit the file lib/names.txt with a the players in your tournament

  2. Run 'rake create_tournament



Now when you view the site you'll see a draw that looks something like this.



For games with 2 players - all first round games and games where the previous round winners have been determined - you'll be able to click the link and set which player won.

This was all very quickly thrown together but if you're interested in hosting any sort of single-elimination tournament check it out on github tournament-draw.

Let me know if you use it and how you'd like to enhance it from the quick application it is today.

Managing the draw of a single-elimination tournament

almost 3 years ago | Alex Rothenberg: Common Sense Software

My company has been running a GO tournament and since its around NCAA March Madness time I thought it'd be easy to find a site to manage the draw. I looked but couldn't find anything that did what I wanted so I wrote my own. Caveat: I just spent a few days on this and it could stand to be enhanced with more features and a rewrite of the ui implementation but I'm open to suggestions if anyone finds this useful.

To use it


  1. Edit the file lib/names.txt with a the players in your tournament

  2. Run 'rake create_tournament



Now when you view the site you'll see a draw that looks something like this.



For games with 2 players - all first round games and games where the previous round winners have been determined - you'll be able to click the link and set which player won.

This was all very quickly thrown together but if you're interested in hosting any sort of single-elimination tournament check it out on github tournament-draw.

Let me know if you use it and how you'd like to enhance it from the quick application it is today.

Link: Search Engine in 200 lines of Ruby Code

almost 3 years ago | Alex Rothenberg: Common Sense Software

I read a very interesting article by someone who works for Yahoo about how he wrote a basic search engine in Ruby in just a few files. I wouldn't use this for a production system for as an example of how search engines work its fascinating and I plan to keep following his site to see how he enhances it

Description: http://blog.saush.com/2009/03/write-an-internet-search-engine-with-200-lines-of-ruby-code/

Source: http://github.com/sausheong/saushengine

... and he's got a really cool css/javascript formatter for the source in his blog. I've gotta learn more about SyntaxHighlighter

Link: Search Engine in 200 lines of Ruby Code

almost 3 years ago | Alex Rothenberg: Common Sense Software

I read a very interesting article by someone who works for Yahoo about how he wrote a basic search engine in Ruby in just a few files. I wouldn't use this for a production system for as an example of how search engines work its fascinating and I plan to keep following his site to see how he enhances it

Description: http://blog.saush.com/2009/03/write-an-internet-search-engine-with-200-lines-of-ruby-code/

Source: http://github.com/sausheong/saushengine

... and he's got a really cool css/javascript formatter for the source in his blog. I've gotta learn more about SyntaxHighlighter

Using scopes in auto_complete plugin

almost 3 years ago | Alex Rothenberg: Common Sense Software

My colleague Pat Shaughnessy has spent a lot of time recently enhancing the auto_complete plugin. I suggest you read his blog posts and check out his fork of auto_complete on github to see the details.

I was reading his latest change to allow filtering of auto complete picklists and really like what he did but thought there was one thing that didn't quite feel right - the fact that you have to mix the application logic to filter the list with the plugin logic to find the list in the block in your controller.

Here's the code Pat wrote in his controller and what I'd like to avoid is having to re-implement the "LOWER(tasks.name) LIKE ?" portion that's already implemented in the filtered_auto_complete_for method of autocomplete.rb in the plugin.

# For task name auto complete, only display tasks
# that belong to the given project: 
filtered_auto_complete_for :task, :name do | find_options, params|
  find_options.merge!(
    {
      :include => :project,
      :conditions => [ "LOWER(tasks.name) LIKE ? AND projects.name = ?",
                       '%' + params['task']['name'].downcase + '%',
                       params['project'] ],
      :order => "tasks.name ASC"
    }
  )
end




I'd like to propose a slight modification so the block you write as part of your application can focus just on the filtering and leave the responsibility for the search with plugin. I'm also proposing we use scopes (which I don't think were around when the original auto_complete plugin was written) to filter and sort the list the plugin. Here is the code I'd like to write in my application.

class ProjectController < ApplicationController
  # For task name auto complete, only display tasks
  # that belong to the given project: 
  filtered_auto_complete_for :task, :name do | list, params |
    list.by_project(params['project']['id'])
  end
end

class Project < ActiveRecord::Base
  named_scope :by_project, 
              lambda {|project_id| {:conditions => {:project_id => project_id} } }
end



We can do this with a simple modification of the plugin implementation of filtered_auto_complete_for

def filtered_auto_complete_for(object, method)
  define_method("auto_complete_for_#{object}_#{method}") do
    find_options = { 
      :conditions => [ "LOWER(#{method}) LIKE ?", '%' +
        params[object][method].downcase + '%' ], 
      :order => "#{method} ASC",
      :limit => 10 }
    @items = object.to_s.camelize.constantize.scoped(find_options)
    @items = yield(@items, params) if block_given?

    render :inline => "<%= auto_complete_result @items, '#{method}' %>"
  end
end



Originally, it used passed the find_options hash to the block and then executed the search in one step as "@items = object.to_s.camelize.constantize.find(:all, find_options)". My change is to rely on ActiveRecord proxy objects to chain criteria together leaving the block independent of the criteria here in the plugin. The best part is you don't have to worry about performance as ActiveRecord will intelligently combine the criteria into a single SQL request. For example searching for tasks that start with 'tas' within project #7


User Load (1.2ms) SELECT * FROM `tasks` WHERE ((`tasks`.`project_id` = '7') AND (LOWER(name) LIKE '%tas%')) ORDER BY name ASC LIMIT 10

Java as a Legacy Language

almost 3 years ago | Alex Rothenberg: Common Sense Software

I came across this article titled Java as Legacy Language today. As an ex-Java guy who is now committed to Ruby I was amused by the title but also think he makes a good point.


One thing is for sure: If you're in the software development business, don't cling to old ways of doing development. And also, don't get too carried away thinking that something like Scrum is going to be the Bandaid that fixes your agility problems, because it may turn out that your main problem is Java itself. Keep an open mind. Try new things. Be ready when the next disruption arrives, or you may find yourself without a chair when the music stops.


I don't think Java is "bad" but I do think it encourages you to write big, complicated applications to solve big, complicated problems. In contrast I find Rails encourages you to realize most of the apps we write are in fact fairly simple and do the same CRUD steps over and over. Having so much support for that built into the framework helps you think about what's "interesting" about your appplication which may be unique but is probably not big.

Add to this how easy it is to test in Rails - tools like RSpec & Shoulda, mocking tools, integration testing and more recently cucumber + webrat all make it easier to practice TDD than not to. While there are tools in Java, the fact that they're harder to use, I believe, means fewer developers will use them and these practices will remain less ingrained in that community.

Using scopes in auto_complete plugin

almost 3 years ago | Alex Rothenberg: Common Sense Software

My colleague Pat Shaughnessy has spent a lot of time recently enhancing the auto_complete plugin. I suggest you read his blog posts and check out his fork of auto_complete on github to see the details.

I was reading his latest change to allow filtering of auto complete picklists and really like what he did but thought there was one thing that didn't quite feel right - the fact that you have to mix the application logic to filter the list with the plugin logic to find the list in the block in your controller.

Here's the code Pat wrote in his controller and what I'd like to avoid is having to re-implement the "LOWER(tasks.name) LIKE ?" portion that's already implemented in the filtered_auto_complete_for method of autocomplete.rb in the plugin.

# For task name auto complete, only display tasks
# that belong to the given project: 
filtered_auto_complete_for :task, :name do | find_options, params|
  find_options.merge!(
    {
      :include => :project,
      :conditions => [ "LOWER(tasks.name) LIKE ? AND projects.name = ?",
                       '%' + params['task']['name'].downcase + '%',
                       params['project'] ],
      :order => "tasks.name ASC"
    }
  )
end




I'd like to propose a slight modification so the block you write as part of your application can focus just on the filtering and leave the responsibility for the search with plugin. I'm also proposing we use scopes (which I don't think were around when the original auto_complete plugin was written) to filter and sort the list the plugin. Here is the code I'd like to write in my application.

class ProjectController < ApplicationController
  # For task name auto complete, only display tasks
  # that belong to the given project: 
  filtered_auto_complete_for :task, :name do | list, params |
    list.by_project(params['project']['id'])
  end
end

class Project < ActiveRecord::Base
  named_scope :by_project, 
              lambda {|project_id| {:conditions => {:project_id => project_id} } }
end



We can do this with a simple modification of the plugin implementation of filtered_auto_complete_for

def filtered_auto_complete_for(object, method)
  define_method("auto_complete_for_#{object}_#{method}") do
    find_options = { 
      :conditions => [ "LOWER(#{method}) LIKE ?", '%' +
        params[object][method].downcase + '%' ], 
      :order => "#{method} ASC",
      :limit => 10 }
    @items = object.to_s.camelize.constantize.scoped(find_options)
    @items = yield(@items, params) if block_given?

    render :inline => "<%= auto_complete_result @items, '#{method}' %>"
  end
end



Originally, it used passed the find_options hash to the block and then executed the search in one step as "@items = object.to_s.camelize.constantize.find(:all, find_options)". My change is to rely on ActiveRecord proxy objects to chain criteria together leaving the block independent of the criteria here in the plugin. The best part is you don't have to worry about performance as ActiveRecord will intelligently combine the criteria into a single SQL request. For example searching for tasks that start with 'tas' within project #7


User Load (1.2ms) SELECT * FROM `tasks` WHERE ((`tasks`.`project_id` = '7') AND (LOWER(name) LIKE '%tas%')) ORDER BY name ASC LIMIT 10

Java as a Legacy Language

almost 3 years ago | Alex Rothenberg: Common Sense Software

I came across this article titled Java as Legacy Language today. As an ex-Java guy who is now committed to Ruby I was amused by the title but also think he makes a good point.


One thing is for sure: If you're in the software development business, don't cling to old ways of doing development. And also, don't get too carried away thinking that something like Scrum is going to be the Bandaid that fixes your agility problems, because it may turn out that your main problem is Java itself. Keep an open mind. Try new things. Be ready when the next disruption arrives, or you may find yourself without a chair when the music stops.


I don't think Java is "bad" but I do think it encourages you to write big, complicated applications to solve big, complicated problems. In contrast I find Rails encourages you to realize most of the apps we write are in fact fairly simple and do the same CRUD steps over and over. Having so much support for that built into the framework helps you think about what's "interesting" about your appplication which may be unique but is probably not big.

Add to this how easy it is to test in Rails - tools like RSpec & Shoulda, mocking tools, integration testing and more recently cucumber + webrat all make it easier to practice TDD than not to. While there are tools in Java, the fact that they're harder to use, I believe, means fewer developers will use them and these practices will remain less ingrained in that community.

Microsoft Office links causing InvalidAuthenticityToken in Rails

almost 3 years ago | Alex Rothenberg: Common Sense Software

I started receiving a lot of error notifications recently from my ExceptionNotfier plugin for an error with ActionController::InvalidAuthenticityToken. It turned out the error was occurring because one of my users was pasting a link to my app in an MS Office document and when Office sees the link it makes a request that Rails could not handle. Here I'll show you a simple fix you can use to avoid these errors with much credit going to an article at
Dealing with Microsoft Office Protocol Discovery in Rails
.

My execptions looked something like this (lots of boring details omitted)


A ActionController::InvalidAuthenticityToken occurred in events#1164:

ActionController::InvalidAuthenticityToken
[RAILS_ROOT]/vendor/rails/actionpack/lib/action_controller/request_forgery_protection.rb:86:in `verify_authenticity_token'

-------------------------------
Environment:
-------------------------------
* HTTP_USER_AGENT : Microsoft Data Access Internet Publishing Provider Protocol Discovery
* REQUEST_METHOD : OPTIONS



The problem is that Rails doesn't understand the method 'OPTIONS' (see rails/actionpack/lib/action_controller/routing.rb

#around line 270 of routing.rb
module ActionController
  module Routing
    HTTP_METHODS = [:get, :head, :post, :put, :delete]
  end
end



Fixing the problem is fairly simple. You insert a before_filter into your application controller to intercept and handle requests with the option method before the rails code realizes it can't handle the request.

class ApplicationController < ActionController::Base
  before_filter :options_for_microsoft_office_protocol_discovery

  ### Lots of code omitted

  def options_for_microsoft_office_protocol_discovery
    render :nothing => true, :status => 200 if request.method == :options
  end
end



Its also easy to write a simple spec in rspec to verify the behavior. There is one trick which is that rails/actionpack/lib/action_controller/test_process.rb defines helper methods for get, post, put, delete & head that we can't use so we need to call the underlying method directly but the signature for that underlying method changed with Rails 2.3 (commit) so depending what version you're using you'll need one of 2 flavors.

# Rails 2.3 and above 
it 'should not throw an exception on OPTIONS request (from ms office protocol discovery)' do
  process '/any/goofy/path', nil, nil, nil, 'OPTIONS'
  response.should be_success
end 

# Rails < 2.3 version
it 'should not throw an exception on OPTIONS request (from ms office protocol discovery)' do
  @request.env['REQUEST_METHOD'] = 'OPTIONS'
  process '/any/goofy/path'
  response.should be_success
end



You can also test from the command line using curl


curl -X OPTIONS http://localhost:3000/



Update: It turns out that Rails stores the acceptable methods in 2 different places actionpack/lib/action_controller/request.rb which does include all of get head put post delete options and also (see rails/actionpack/lib/action_controller/routing.rb which only includes :get, :head, :post, :put, :delete (options is missing). This means this fix will only work for OPTIONS requests and not any other type as ActionController::Request request_method will throw an exception before getting to the filter code above.

Microsoft Office links causing InvalidAuthenticityToken in Rails

almost 3 years ago | Alex Rothenberg: Common Sense Software

I started receiving a lot of error notifications recently from my ExceptionNotfier plugin for an error with ActionController::InvalidAuthenticityToken. It turned out the error was occurring because one of my users was pasting a link to my app in an MS Office document and when Office sees the link it makes a request that Rails could not handle. Here I'll show you a simple fix you can use to avoid these errors with much credit going to an article at
Dealing with Microsoft Office Protocol Discovery in Rails
.

My execptions looked something like this (lots of boring details omitted)


A ActionController::InvalidAuthenticityToken occurred in events#1164:

ActionController::InvalidAuthenticityToken
[RAILS_ROOT]/vendor/rails/actionpack/lib/action_controller/request_forgery_protection.rb:86:in `verify_authenticity_token'

-------------------------------
Environment:
-------------------------------
* HTTP_USER_AGENT : Microsoft Data Access Internet Publishing Provider Protocol Discovery
* REQUEST_METHOD : OPTIONS



The problem is that Rails doesn't understand the method 'OPTIONS' (see rails/actionpack/lib/action_controller/routing.rb

#around line 270 of routing.rb
module ActionController
  module Routing
    HTTP_METHODS = [:get, :head, :post, :put, :delete]
  end
end



Fixing the problem is fairly simple. You insert a before_filter into your application controller to intercept and handle requests with the option method before the rails code realizes it can't handle the request.

class ApplicationController < ActionController::Base
  before_filter :options_for_microsoft_office_protocol_discovery

  ### Lots of code omitted

  def options_for_microsoft_office_protocol_discovery
    render :nothing => true, :status => 200 if request.method == :options
  end
end



Its also easy to write a simple spec in rspec to verify the behavior. There is one trick which is that rails/actionpack/lib/action_controller/test_process.rb defines helper methods for get, post, put, delete & head that we can't use so we need to call the underlying method directly but the signature for that underlying method changed with Rails 2.3 (commit) so depending what version you're using you'll need one of 2 flavors.

# Rails 2.3 and above 
it 'should not throw an exception on OPTIONS request (from ms office protocol discovery)' do
  process '/any/goofy/path', nil, nil, nil, 'OPTIONS'
  response.should be_success
end 

# Rails < 2.3 version
it 'should not throw an exception on OPTIONS request (from ms office protocol discovery)' do
  @request.env['REQUEST_METHOD'] = 'OPTIONS'
  process '/any/goofy/path'
  response.should be_success
end



You can also test from the command line using curl


curl -X OPTIONS http://localhost:3000/



Update: It turns out that Rails stores the acceptable methods in 2 different places actionpack/lib/action_controller/request.rb which does include all of get head put post delete options and also (see rails/actionpack/lib/action_controller/routing.rb which only includes :get, :head, :post, :put, :delete (options is missing). This means this fix will only work for OPTIONS requests and not any other type as ActionController::Request request_method will throw an exception before getting to the filter code above.

Investigating how Symbol to_proc works

about 3 years ago | Alex Rothenberg: Common Sense Software

One of the things I love about Ruby is how expressive it is and how with open classes it can be optimized to become even more expressive. Since I started using Ruby I don't think I've written a single for or while loop - something I couldn't have imagined saying with any other language! Of course I do this by using iterators and writing code like
user_names = User.all.collect {|user| user.name }

I recently started discovered I could write the same thing even more concisely (as long as I'm using Rails or Ruby 1.9)
user_names = User.all.collect(&:name)

I decided to investigate how this works.

First I found some good posts by Prag Dave and Ryan Bates and at InfoQ. This helped but I still didn't understand it all so decided to dig further.

First I took a look at how Rails extends Symbol

unless :to_proc.respond_to?(:to_proc)
  class Symbol
    # Turns the symbol into a simple proc, which is especially useful for enumerations. Examples:
    #
    #   # The same as people.collect { |p| p.name }
    #   people.collect(&:name)
    #
    #   # The same as people.select { |p| p.manager? }.collect { |p| p.salary }
    #   people.select(&:manager?).collect(&:salary)
    def to_proc
      Proc.new { |*args| args.shift.__send__(self, *args) }
    end
  end
end


So they defined the to_proc method on symbol and that means the new code will be called when we write &:name because it magically gets transformed into :name.to_proc. I learned something but still needed to learn more to understand how it all works.

Why does the & cause Ruby to call to_proc? I knew that & in the last parameter declaration will pass a provided block as a parameter but this seems to be doing the reverse. Calling a method as an argument but having it interpreted as a block. I tried a couple of experiments in irb

def was_block_given?
  block_given?
end

# As expected
was_block_given? {}
=>  true 
# Passing a proc is not the same as having a block
was_block_given? Proc.new{}
ArgumentError: wrong number of arguments (1 for 0)
  from (irb):195:in 'was_block_given?'
  from (irb):195
  
# Prefixing the proc with an &amp; makes it like a block
was_block_given? &amp;Proc.new{}
=> true

It was not all as I expected but some reading through the PickAxe book led me to a better understanding. I found this paragraph in the Calling A Method section (page 115 in my copy)

If the last argument to a method is preceded by an ampersand, Ruby assumes that it is a Proc object.
It removes it from the parameter list, converts the Proc object into a block, and associates it with the method.


Ok so now I know why when Ruby sees User.all.collect(&:name) it invokes the collect method with name.to_proc as a block. Next, it was time to figure out why the code Rails put in the to_proc method worked. I took a look at the Rubinius implementation Enumerable

def collect
  ary = []
  if block_given?
    each { |o| ary << yield(o) }
  else
    each { |o| ary << o }
  end
  ary
end

Again I decided to experiment with irb to see what each part of the to_proc implementation was doing. First I redefined the Symbol to_proc again with a puts so I could confirm what was going on.

class Symbol
  # Turns the symbol into a simple proc, which is especially useful for enumerations. Examples:
  #
  #   # The same as people.collect { |p| p.name }
  #   people.collect(&:name)
  #
  #   # The same as people.select { |p| p.manager? }.collect { |p| p.salary }
  #   people.select(&:manager?).collect(&:salary)
  def to_proc
    Proc.new do |*args| 
      puts "to_proc args: #{args.inspect}"
      args_shift = args.shift
      puts "to_proc: #{args_shift.inspect}.__send__(#{self.inspect}, *#{args.inspect})"
      result = args_shift.__send__(self, *args) 
      puts "to_proc result: #{result.inspect}"
      result
    end
  end
end

# Make the call and see what happens
[1].collect(&:to_s)
# to_proc args: [1]
# to_proc: 1.__send__(:to_s, *[])
# to_proc result: "1"
# => ["1"]

Its brute force but tells us everything we need to know. As expected collect yields to our proc/block with the element in a variable length argument [1], it extracts the 1 and sends it the to_s method with no arguments returning the string "1". At this point I think I understand how it all works and decide to confirm by running a few more (more complicated) tests in irb


It all works as expected and I decide I know as much as I need to about this and call it a day.

So why did I bother figuring all this out and then writing it up? Mostly because I didn't know how it worked and thought there was some 'magic' going on. I could have continued using this feature without understanding how it worked but now that I understand how it works if some need ever arises for me to do some similar magic I know how to go about it. As for writing it up I hope someone else may read this and find it useful but by I increased my own understanding through the act of writing.

Investigating how Symbol to_proc works

about 3 years ago | Alex Rothenberg: Common Sense Software

One of the things I love about Ruby is how expressive it is and how with open classes it can be optimized to become even more expressive. Since I started using Ruby I don't think I've written a single for or while loop - something I couldn't have imagined saying with any other language! Of course I do this by using iterators and writing code like
user_names = User.all.collect {|user| user.name }

I recently started discovered I could write the same thing even more concisely (as long as I'm using Rails or Ruby 1.9)
user_names = User.all.collect(&:name)

I decided to investigate how this works.

First I found some good posts by Prag Dave and Ryan Bates and at InfoQ. This helped but I still didn't understand it all so decided to dig further.

First I took a look at how Rails extends Symbol

unless :to_proc.respond_to?(:to_proc)
  class Symbol
    # Turns the symbol into a simple proc, which is especially useful for enumerations. Examples:
    #
    #   # The same as people.collect { |p| p.name }
    #   people.collect(&:name)
    #
    #   # The same as people.select { |p| p.manager? }.collect { |p| p.salary }
    #   people.select(&:manager?).collect(&:salary)
    def to_proc
      Proc.new { |*args| args.shift.__send__(self, *args) }
    end
  end
end


So they defined the to_proc method on symbol and that means the new code will be called when we write &:name because it magically gets transformed into :name.to_proc. I learned something but still needed to learn more to understand how it all works.

Why does the & cause Ruby to call to_proc? I knew that & in the last parameter declaration will pass a provided block as a parameter but this seems to be doing the reverse. Calling a method as an argument but having it interpreted as a block. I tried a couple of experiments in irb

def was_block_given?
  block_given?
end

# As expected
was_block_given? {}
=>  true 
# Passing a proc is not the same as having a block
was_block_given? Proc.new{}
ArgumentError: wrong number of arguments (1 for 0)
  from (irb):195:in 'was_block_given?'
  from (irb):195
  
# Prefixing the proc with an &amp; makes it like a block
was_block_given? &amp;Proc.new{}
=> true

It was not all as I expected but some reading through the PickAxe book led me to a better understanding. I found this paragraph in the Calling A Method section (page 115 in my copy)

If the last argument to a method is preceded by an ampersand, Ruby assumes that it is a Proc object.
It removes it from the parameter list, converts the Proc object into a block, and associates it with the method.


Ok so now I know why when Ruby sees User.all.collect(&:name) it invokes the collect method with name.to_proc as a block. Next, it was time to figure out why the code Rails put in the to_proc method worked. I took a look at the Rubinius implementation Enumerable

def collect
  ary = []
  if block_given?
    each { |o| ary << yield(o) }
  else
    each { |o| ary << o }
  end
  ary
end

Again I decided to experiment with irb to see what each part of the to_proc implementation was doing. First I redefined the Symbol to_proc again with a puts so I could confirm what was going on.

class Symbol
  # Turns the symbol into a simple proc, which is especially useful for enumerations. Examples:
  #
  #   # The same as people.collect { |p| p.name }
  #   people.collect(&:name)
  #
  #   # The same as people.select { |p| p.manager? }.collect { |p| p.salary }
  #   people.select(&:manager?).collect(&:salary)
  def to_proc
    Proc.new do |*args| 
      puts "to_proc args: #{args.inspect}"
      args_shift = args.shift
      puts "to_proc: #{args_shift.inspect}.__send__(#{self.inspect}, *#{args.inspect})"
      result = args_shift.__send__(self, *args) 
      puts "to_proc result: #{result.inspect}"
      result
    end
  end
end

# Make the call and see what happens
[1].collect(&:to_s)
# to_proc args: [1]
# to_proc: 1.__send__(:to_s, *[])
# to_proc result: "1"
# => ["1"]

Its brute force but tells us everything we need to know. As expected collect yields to our proc/block with the element in a variable length argument [1], it extracts the 1 and sends it the to_s method with no arguments returning the string "1". At this point I think I understand how it all works and decide to confirm by running a few more (more complicated) tests in irb


It all works as expected and I decide I know as much as I need to about this and call it a day.

So why did I bother figuring all this out and then writing it up? Mostly because I didn't know how it worked and thought there was some 'magic' going on. I could have continued using this feature without understanding how it worked but now that I understand how it works if some need ever arises for me to do some similar magic I know how to go about it. As for writing it up I hope someone else may read this and find it useful but by I increased my own understanding through the act of writing.

Maintaining your technical chops is a full time job

about 3 years ago | Alex Rothenberg: Common Sense Software

Great quote from Uncle Bob (near the bottom of the post)

I think that maintaining your technical chops is a full time job. For that reason I have avoided becoming a business wonk. I hire people to do that for me so I can keep my technical skills as sharp as possible and remain relevant to my profession. I don’t believe I can offer technical advice unless I am living that technical advice.

Maintaining your technical chops is a full time job

about 3 years ago | Alex Rothenberg: Common Sense Software

Great quote from Uncle Bob (near the bottom of the post)

I think that maintaining your technical chops is a full time job. For that reason I have avoided becoming a business wonk. I hire people to do that for me so I can keep my technical skills as sharp as possible and remain relevant to my profession. I don’t believe I can offer technical advice unless I am living that technical advice.

Never sacrifice quality for speed!

about 3 years ago | Alex Rothenberg: Common Sense Software

I just read two really good articles by Ron Jeffries and Uncle Bob about why sacrificing quality to go faster is always a bad idea.

This is very relevant to me now as I'm working with a sponsor now who thinks that by pushing harder and 'doing it in parallel' he can get everything he wants by the date he wants avoiding any hard decisions involving tradeoffs between scope and date. I have worked on many teams over the years that could speed up without sacrificing quality by focusing on the right things basically by emphasizing working software over high ceremony (pretty much straight from the Agile Manifesto). However the idea that you can get something out the door quickly without "wasting your time on quality" to me means that the person you're talking to doesn't understand quality. They think quality means those silly engineering things developers spend their time on rather than software that works the way you want and can be reliably enhanced later.

Somehow we as technologists need to do a better job of explaining to our business sponsors that "quality" is not a technical term!

Never sacrifice quality for speed!

about 3 years ago | Alex Rothenberg: Common Sense Software

I just read two really good articles by Ron Jeffries and Uncle Bob about why sacrificing quality to go faster is always a bad idea.

This is very relevant to me now as I'm working with a sponsor now who thinks that by pushing harder and 'doing it in parallel' he can get everything he wants by the date he wants avoiding any hard decisions involving tradeoffs between scope and date. I have worked on many teams over the years that could speed up without sacrificing quality by focusing on the right things basically by emphasizing working software over high ceremony (pretty much straight from the Agile Manifesto). However the idea that you can get something out the door quickly without "wasting your time on quality" to me means that the person you're talking to doesn't understand quality. They think quality means those silly engineering things developers spend their time on rather than software that works the way you want and can be reliably enhanced later.

Somehow we as technologists need to do a better job of explaining to our business sponsors that "quality" is not a technical term!

How I refine a test spec while writing getting it to green

about 3 years ago | Alex Rothenberg: Common Sense Software

In most of the examples I've read on TDD they show the Red-Green-Refactor cycle as


  1. Write a failing test

  2. Write just enough code to make the test pass

  3. Refactor

  4. Repeat



I absolutely do believe in this cycle and live it every day however I think there's a small detail that differs from what I do. When I move from step 1 to step 2, I keep my test window open and will switch back and forth between the test and code windows and refining the test while writing the code that makes the test pass. Its only as I write the code that I realize what calls I need to mock out in the test which also adds additional expectations to my test. I'm sure this is how most people work but I've never seen it written up so I'm going to try going through an example here.

Let's say I have a social networking application with a page showing friend requests where you can accept or reject each request. Something like this


Friend Requests:

Accept request from Pat?
Accept request from Gourav?



Now I'm ready to implement an action in the Friend Controller that receives a posted form with a list of friend requests to accept and reject. I might first go into my friends_controller_spec.rb and write something like


describe FriendController do
it 'should approve one friend requests' do
post :update_requests, :approved_requests => ['123']
response.should be_redirect
response.should redirect_to(:back)
flash[:notice].should == "Friend requests approved."
end
end



Now I have a failing test so I start implementing the code


class FriendController < ApplicationController
def update_requests
FriendRequest.approve(params[:approve_requests])
flash[:notice] = "User access changes saved."
redirect_to :back
end
end



I'm done but my test is still failing...why? I haven't added the approve method to my FriendRequest model yet. I could go ahead and implement that method to get this test to pass but I don't want to do that because then this test would be testing more than just one unit. This is a unit test for the update_requests method of my FriendController and should be isolated from bugs in the rest of my application. Time to go back to the spec and add a mock to isolate this spec from the model and impose another expectation on my spec (that the method be called with the correct argument).


describe FriendController do
it 'should approve one friend requests' do
FriendRequest.expects(:approve).with(['123'])
request.env['HTTP_REFERER'] = "http://some.site.com"

post :update_requests, :approved_requests => ['123']
response.should be_redirect
response.should redirect_to(:back)
flash[:notice].should == "Friend requests approved."
end
end



Now my spec passes and I can do my refactoring then go back to the beginning and write my next expectation.

The point of this example is that when I first wrote the spec I hadn't yet thought about how I would implement update_requests so did not yet know that I would need to mock FriendRequest.approve. I do this all the time and typically find myself go back and forth between the code and spec many times with the spec telling me what code I need to write and the code telling me how to refine the spec.

One last point (and a teaser for a future post). You have probably noticed that although my spec passes if I try to run the application it will not work because I still haven't written the FriendRequest.approve method. In a future post I hope to discuss how to interleave integration testing into this process to ensure that the classes we're writing in isolation also integrate so that the application satisfies its business function.

How I refine a test spec while writing getting it to green

about 3 years ago | Alex Rothenberg: Common Sense Software

In most of the examples I've read on TDD they show the Red-Green-Refactor cycle as


  1. Write a failing test

  2. Write just enough code to make the test pass

  3. Refactor

  4. Repeat



I absolutely do believe in this cycle and live it every day however I think there's a small detail that differs from what I do. When I move from step 1 to step 2, I keep my test window open and will switch back and forth between the test and code windows and refining the test while writing the code that makes the test pass. Its only as I write the code that I realize what calls I need to mock out in the test which also adds additional expectations to my test. I'm sure this is how most people work but I've never seen it written up so I'm going to try going through an example here.

Let's say I have a social networking application with a page showing friend requests where you can accept or reject each request. Something like this


Friend Requests:

Accept request from Pat?
Accept request from Gourav?



Now I'm ready to implement an action in the Friend Controller that receives a posted form with a list of friend requests to accept and reject. I might first go into my friends_controller_spec.rb and write something like


describe FriendController do
it 'should approve one friend requests' do
post :update_requests, :approved_requests => ['123']
response.should be_redirect
response.should redirect_to(:back)
flash[:notice].should == "Friend requests approved."
end
end



Now I have a failing test so I start implementing the code


class FriendController < ApplicationController
def update_requests
FriendRequest.approve(params[:approve_requests])
flash[:notice] = "User access changes saved."
redirect_to :back
end
end



I'm done but my test is still failing...why? I haven't added the approve method to my FriendRequest model yet. I could go ahead and implement that method to get this test to pass but I don't want to do that because then this test would be testing more than just one unit. This is a unit test for the update_requests method of my FriendController and should be isolated from bugs in the rest of my application. Time to go back to the spec and add a mock to isolate this spec from the model and impose another expectation on my spec (that the method be called with the correct argument).


describe FriendController do
it 'should approve one friend requests' do
FriendRequest.expects(:approve).with(['123'])
request.env['HTTP_REFERER'] = "http://some.site.com"

post :update_requests, :approved_requests => ['123']
response.should be_redirect
response.should redirect_to(:back)
flash[:notice].should == "Friend requests approved."
end
end



Now my spec passes and I can do my refactoring then go back to the beginning and write my next expectation.

The point of this example is that when I first wrote the spec I hadn't yet thought about how I would implement update_requests so did not yet know that I would need to mock FriendRequest.approve. I do this all the time and typically find myself go back and forth between the code and spec many times with the spec telling me what code I need to write and the code telling me how to refine the spec.

One last point (and a teaser for a future post). You have probably noticed that although my spec passes if I try to run the application it will not work because I still haven't written the FriendRequest.approve method. In a future post I hope to discuss how to interleave integration testing into this process to ensure that the classes we're writing in isolation also integrate so that the application satisfies its business function.

How I use TDD for Integration and Isolation testing

about 3 years ago | Alex Rothenberg: Common Sense Software

I read an interesting post today by David Chelimsky who wrote RSpec A case against a case against mocking and stubbing. Its about mocking in testing and isolated vs integrated tests. I liked it all but what I particularly liked is how it describes the process of outside-in development.

To quote:


  1. Write scenarios in plain text with cucumber (driven by user stories, organized in features).

  2. Write the code for a step (or part of a step), run the feature, and observe the failure.

  3. Optionally (yes, it depends – and why is the topic for another blog) drive out a view with a view spec. When I say “drive out,” I mean a very granular Red/Green/Refactor cycle that only involves this view, and only enough of this view to support satisfaction of the failing step in the cucumber feature.

  4. Drive out a controller action using the same, granular Red/Green/Refactor cycle. And it may not be the entire controller action I think I want, covering all the cases I think I want. Just enough to support satisfaction of the failing step.

  5. Drive out the parts of the model that I need to satisfy the failing step, using the same granular R/G/R process.

  6. Run the cucumber feature and assess where I am.



In my projects I think I'm pretty good at following 3,4 &5 but I've struggled with steps 1 & 6. With the advent of cucumber I think now is the time to work on those other steps. I'll let you know how it goes in the New Year.

How I use TDD for Integration and Isolation testing

about 3 years ago | Alex Rothenberg: Common Sense Software

I read an interesting post today by David Chelimsky who wrote RSpec A case against a case against mocking and stubbing. Its about mocking in testing and isolated vs integrated tests. I liked it all but what I particularly liked is how it describes the process of outside-in development.

To quote:


  1. Write scenarios in plain text with cucumber (driven by user stories, organized in features).

  2. Write the code for a step (or part of a step), run the feature, and observe the failure.

  3. Optionally (yes, it depends – and why is the topic for another blog) drive out a view with a view spec. When I say “drive out,” I mean a very granular Red/Green/Refactor cycle that only involves this view, and only enough of this view to support satisfaction of the failing step in the cucumber feature.

  4. Drive out a controller action using the same, granular Red/Green/Refactor cycle. And it may not be the entire controller action I think I want, covering all the cases I think I want. Just enough to support satisfaction of the failing step.

  5. Drive out the parts of the model that I need to satisfy the failing step, using the same granular R/G/R process.

  6. Run the cucumber feature and assess where I am.



In my projects I think I'm pretty good at following 3,4 &5 but I've struggled with steps 1 & 6. With the advent of cucumber I think now is the time to work on those other steps. I'll let you know how it goes in the New Year.

Bug/patch with rspec-rails and helper instance variables

about 3 years ago | Alex Rothenberg: Common Sense Software

I finally got around to upgrading my version of rspec-rails from one that's almost a year old and came across an issue with the way implicit module inclusion is handled.

If you have a handler that uses memoization to cache some information in instance variables such as this (I'm not sure if this is a smell but my project has some examples of it)


module UsersHelper
def all_users
@users ||= User.find(:all)
end
end



You would expect this spec to work


describe UsersHelper do
it "should find all users" do
User.expects(:find).with(:all).returns(result=mock)
helper.all_users.should == result
end
end



Sometimes it does but if any other spec has called helper.all_users previously it will fail as the memoized @users variable is not nil so the collection is reused.

I have submitted a lighthouse patch to rspec-rails that fixes the problem so hopefully they will agree to fix it soon. Until then patch is available on github in my fork or rspec-rails if you want to use it.

Knows which specs are slowing down my spec suite

about 3 years ago | Alex Rothenberg: Common Sense Software

I often watch my specs run with the ........'s marching across my terminal and sometimes it seems to pause as it hits a particularly slow one. I figure there's some poorly written spec that's slowing my entire suite down and I should figure out what it is and fix it. But I rarely (if ever) take the time to track down the offender so never fix it. I just discovered that the clever folks writing rspec have taken away my excuse for laziness. I can tell it to report on the slowest specs by putting a line in my spec.opts file telling it to use the ProfileFormatter (see the first line below)


--format profile
--colour
--loadby mtime
--reverse



For example running the specs on my current project I get this output.


Profiling enabled.
........(a bunch more dots)

Top 10 slowest examples:
2.4908950 WikiContent should find all activity for a user
0.8061750 WikiContent should find recent activity for a user in last 15 days
0.2438460 WikiContent::Version should allow limit on results from finding recent activity
0.1821710 User should have a list of owned content
0.1700670 WikiContent should find results if access type is not same as role of the user but role of user is 'Leader'
0.1593900 ContentsController allows PUT request to 'update' action for 'member' role
0.1514150 Search::SearchResponse gets the wiki content id of an asset from the xml response
0.1470310 Content should always allow leaders to see all pages
0.1394920 Page should delete all comments when destroyed
0.1370330 ContentsController should save content and create the uploaded asset on successful POST to the 'create' action



Over the next few days I will find time to take a look at 'WikiContent find all activity' and 'WikiContent find recent activity' specs as they are the two clear outliers.

I now have no excuse to not optimize my slowest specs because I didn't know which they were!

Bug/patch with rspec-rails and helper instance variables

about 3 years ago | Alex Rothenberg: Common Sense Software

I finally got around to upgrading my version of rspec-rails from one that's almost a year old and came across an issue with the way implicit module inclusion is handled.

If you have a handler that uses memoization to cache some information in instance variables such as this (I'm not sure if this is a smell but my project has some examples of it)


module UsersHelper
def all_users
@users ||= User.find(:all)
end
end



You would expect this spec to work


describe UsersHelper do
it "should find all users" do
User.expects(:find).with(:all).returns(result=mock)
helper.all_users.should == result
end
end



Sometimes it does but if any other spec has called helper.all_users previously it will fail as the memoized @users variable is not nil so the collection is reused.

I have submitted a lighthouse patch to rspec-rails that fixes the problem so hopefully they will agree to fix it soon. Until then patch is available on github in my fork or rspec-rails if you want to use it.

Knows which specs are slowing down my spec suite

about 3 years ago | Alex Rothenberg: Common Sense Software

I often watch my specs run with the ........'s marching across my terminal and sometimes it seems to pause as it hits a particularly slow one. I figure there's some poorly written spec that's slowing my entire suite down and I should figure out what it is and fix it. But I rarely (if ever) take the time to track down the offender so never fix it. I just discovered that the clever folks writing rspec have taken away my excuse for laziness. I can tell it to report on the slowest specs by putting a line in my spec.opts file telling it to use the ProfileFormatter (see the first line below)


--format profile
--colour
--loadby mtime
--reverse



For example running the specs on my current project I get this output.


Profiling enabled.
........(a bunch more dots)

Top 10 slowest examples:
2.4908950 WikiContent should find all activity for a user
0.8061750 WikiContent should find recent activity for a user in last 15 days
0.2438460 WikiContent::Version should allow limit on results from finding recent activity
0.1821710 User should have a list of owned content
0.1700670 WikiContent should find results if access type is not same as role of the user but role of user is 'Leader'
0.1593900 ContentsController allows PUT request to 'update' action for 'member' role
0.1514150 Search::SearchResponse gets the wiki content id of an asset from the xml response
0.1470310 Content should always allow leaders to see all pages
0.1394920 Page should delete all comments when destroyed
0.1370330 ContentsController should save content and create the uploaded asset on successful POST to the 'create' action



Over the next few days I will find time to take a look at 'WikiContent find all activity' and 'WikiContent find recent activity' specs as they are the two clear outliers.

I now have no excuse to not optimize my slowest specs because I didn't know which they were!

Applications usually use their own data

about 3 years ago | Alex Rothenberg: Common Sense Software

I just read a very interesting post by Martin Fowler called Database Thaw. He talks about various database technologies and what the future might hold for object or relational databases but what caught my interest was his discussion of application and database integration patterns.

He says "For many organizations today, the primary pattern for integration is Shared Database Integration - where multiple applications are integrated by all using a common database." This is certainly true where I work. I've even heard it said that we have one gigantic application since all (or most) share the same underlying database so cannot be updated independently.

He also talks about Integration Databases which store data for multiple applications and Application Databases which are controlled and accessed by a single application.

Historically my company has moved toward the integration database pattern often building complex service layers to enable shared access to the data on the assumption that if you build it they will come the data would be reused if it was easy enough. However I believe that its only a very small subset of our data that is truly common and many applications want to use. For that small subset a shared database with a service layer is probably a good design. The vast majority is private to an application and by designing it into a separate application database can evolve to best meet the needs of the application without concern for a more general service that in our case is more hindrance than benefit.

One important thing to remember about an application database is that the data exists to meet the needs of the application so a design that allows the database to best meet the application's needs will be to everyone's benefit. Finally database technologies are very complex and do a lot so it is probably wise to have a good database developer on your team to make sure you're using the database appropriately but they should be a part of your team not on a separate database team as I've seen in the past - particularly on Java/Oracle projects I've been a part of.

Thoughts?

Applications usually use their own data

about 3 years ago | Alex Rothenberg: Common Sense Software

I just read a very interesting post by Martin Fowler called Database Thaw. He talks about various database technologies and what the future might hold for object or relational databases but what caught my interest was his discussion of application and database integration patterns.

He says "For many organizations today, the primary pattern for integration is Shared Database Integration - where multiple applications are integrated by all using a common database." This is certainly true where I work. I've even heard it said that we have one gigantic application since all (or most) share the same underlying database so cannot be updated independently.

He also talks about Integration Databases which store data for multiple applications and Application Databases which are controlled and accessed by a single application.

Historically my company has moved toward the integration database pattern often building complex service layers to enable shared access to the data on the assumption that if you build it they will come the data would be reused if it was easy enough. However I believe that its only a very small subset of our data that is truly common and many applications want to use. For that small subset a shared database with a service layer is probably a good design. The vast majority is private to an application and by designing it into a separate application database can evolve to best meet the needs of the application without concern for a more general service that in our case is more hindrance than benefit.

One important thing to remember about an application database is that the data exists to meet the needs of the application so a design that allows the database to best meet the application's needs will be to everyone's benefit. Finally database technologies are very complex and do a lot so it is probably wise to have a good database developer on your team to make sure you're using the database appropriately but they should be a part of your team not on a separate database team as I've seen in the past - particularly on Java/Oracle projects I've been a part of.

Thoughts?

Silly site - Netdisaster ... Destroy the Web

over 3 years ago | Alex Rothenberg: Common Sense Software

My kids just showed me a silly site http://www.netdisaster.com/ I had to share.

Silly site - Netdisaster ... Destroy the Web

over 3 years ago | Alex Rothenberg: Common Sense Software

My kids just showed me a silly site http://www.netdisaster.com/ I had to share.

Buy services and Build websites

over 3 years ago | Alex Rothenberg: Common Sense Software

Where I work there's recently been a lot of talk about how to make the decision whether to buy or build software. I've recently concluded that while you can buy a commoditized service you can not buy a website!

What I've seen is people thinking we could "buy" (or download open source) a site but then realize they want to change the branding, adapt the terminology to match our business. combine it with another product or add one extra piece of functionality that the existing site doesn't have. Whatever the APIs provided by the product you're left with custom software written on top of the product you bought. I've seen this with big commercial products like Documentum and open source products like Drupal. I now believe that there's no such thing as "a little" customization and you can only buy a site if you want 0 customization (applying a skin or inserting your logos are acceptable).

This does not mean everyone should develop custom software from scratch. There are tons of opportunities to buy commoditized services and write just a little custom code to glue them together. This allows you to focus on the aspects of your site that are business differentiators (the branding, the terminology, your workflows, your business rules, etc) without fighting the vendor's idea of those things. You don't have to write too much code as what we continually revise upward what we consider a commoditized service (i.e. it used to be document storage and is becoming document management with workflow and security).

The process of creating a blog aggregator as I described in my last post I built a Blog Aggregator showed me how true this is. I used Atom and RSS parsing and generating service which left me just to build a small site with the UI, security and glue holding it all together. Its just a small example but as I look around github I'm struck that the most interesting packages are plugins and gems for others to use and the less interesting ones are complete sites. This may be a no brainer for the Open Source community but is something that I am just realizing and I think that enterprises and commercial vendors still need to learn.

Buy services and Build websites

over 3 years ago | Alex Rothenberg: Common Sense Software

Where I work there's recently been a lot of talk about how to make the decision whether to buy or build software. I've recently concluded that while you can buy a commoditized service you can not buy a website!

What I've seen is people thinking we could "buy" (or download open source) a site but then realize they want to change the branding, adapt the terminology to match our business. combine it with another product or add one extra piece of functionality that the existing site doesn't have. Whatever the APIs provided by the product you're left with custom software written on top of the product you bought. I've seen this with big commercial products like Documentum and open source products like Drupal. I now believe that there's no such thing as "a little" customization and you can only buy a site if you want 0 customization (applying a skin or inserting your logos are acceptable).

This does not mean everyone should develop custom software from scratch. There are tons of opportunities to buy commoditized services and write just a little custom code to glue them together. This allows you to focus on the aspects of your site that are business differentiators (the branding, the terminology, your workflows, your business rules, etc) without fighting the vendor's idea of those things. You don't have to write too much code as what we continually revise upward what we consider a commoditized service (i.e. it used to be document storage and is becoming document management with workflow and security).

The process of creating a blog aggregator as I described in my last post I built a Blog Aggregator showed me how true this is. I used Atom and RSS parsing and generating service which left me just to build a small site with the UI, security and glue holding it all together. Its just a small example but as I look around github I'm struck that the most interesting packages are plugins and gems for others to use and the less interesting ones are complete sites. This may be a no brainer for the Open Source community but is something that I am just realizing and I think that enterprises and commercial vendors still need to learn.

I built a blog aggregator - waywework.it

over 3 years ago | Alex Rothenberg: Common Sense Software

I've been spending some time recently putting together a blog aggregator site for some of the folks I work with. Its now up and running at http://waywework.it. I hope this will be an interesting place to share our public community and as one of my colleagues said "this keeps my Google Reader much neater".

Today I'd like to talk about the code running this site which is posted and available on github at http://github.com/alexrothenberg/waywework.

I started thinking I would use an existing aggregator site and just apply my skin but when I did a quick search on github I most of the hard work existed in atom and rss gems and plugins and I wanted to take advantage of the just released Rails 2.2 so I decided to build my own. This turned out to be not too much work. Today I'd like to talk about how I put this together.

First I created my project with some scaffolding for feeds which would have_many posts

class Feed < ActiveRecord::Base
  has_many :posts, :dependent => :delete_all
end

class Post < ActiveRecord::Base
  belongs_to :feed
end



I soon found the atom gem and rss parser built into ruby. Using them was a piece of cake as all I had to do was create a method to call each one in my Feed model

class Feed < ActiveRecord::Base
  def get_posts_from_atom atom_xml
    feed = Atom::Feed.new(atom_xml)
    feed.entries.each { |entry|
      link = entry.links.detect {|l| l.rel == 'alternate'}
      create_post(:contents=>entry.content.value, :url=>link.href, :title=>entry.title, 
                  :published=>entry.published.to_s(:db), :updated=>entry.updated.to_s(:db))
    }
    return !feed.entries.blank?
  end  
  
  def get_posts_from_rss rss_xml
    rss = RSS::Parser.parse(rss_xml, false)
    rss.items.each { |entry|
      create_post(:contents=>entry.description, :url=>entry.link, :title=>entry.title, 
                  :published=>entry.date.to_formatted_s(:db), :updated=>entry.date.to_formatted_s(:db))
    }
    return !rss.items.blank?
  end
end



Of course I had to create the glue wrapping it all together. A rake task to be call on a schedule

namespace :feeds do
  desc "Load the feeds"
  task :populate => :environment do
    feeds = Feed.all
    feeds.each do |feed|
      feed.get_latest
    end
  end
end



and the logic to load the feed, parse it and update the posts.

class Feed < ActiveRecord::Base
  def get_latest
    puts "getting feed for #{name}"
    xml = get_feed
    got_atom_posts = get_posts_from_atom xml
    get_posts_from_rss xml unless got_atom_posts
  end

  def get_feed
    uri = URI.parse(feed_url)
    uri.read
  end
  
  def create_post params
    params.merge!(:feed_id=>id) 
    existing_post = Post.find_by_url(params[:url])
    if existing_post
      existing_post.update_attributes(params)
    else
      Post.create(params) 
    end
  end
end



The next step was to publish an atom feed of my site. Again there was a plugin atom_feed_helperwaiting to help me. I installed the plugin and created a view builder

atom_feed(:url => atom_feed_url) do |feed|
  feed.title("WayWeWork")
  feed.updated(@posts.first.published)

  for post in @posts
    feed.entry(post, :url=>post.url, :published=>post.published, :updated=>post.updated) do |entry|
      entry.title("#{post.feed.author}: #{post.title}")
      entry.content(post.contents, :type => 'html')
    end
  end
end


This was all so easy I had hardly done anything other than glue these plugins together. Now I finished up with a few bells and whistles.

I added a who's talking and archive by date section to my homepage that I called from my controller like this

class PostsController < ApplicationController
  @active_feeds = Feed.by_author
  @activity_by_date = Post.activity_by_date
end



I added security to restrict who can administer feeds

class FeedsController < ApplicationController
  before_filter :authenticate

protected
  def authenticate
     authenticate_or_request_with_http_basic do | user_name, password|
       username = YAML::load_file(File.join(RAILS_ROOT, %w[config password.yml]))['username']
       pwd = YAML::load_file(File.join(RAILS_ROOT, %w[config password.yml]))['password']
      user_name == username && password == pwd
    end
  end
end



For the UI I am somewhat graphically challenged so got some help. For this github was very cool as I could add lessallan as a collaborator and he could check in his changes so they just appeared!

Finally a little work with capistrano (mostly just creating a Capfile) and I could deploy!

Overall I spent a few days and now have a site that does exactly what I want. Where most of the code I wrote is specific to my site and the general purpose plumbing was downloaded. I'm very pleased with the availability of plugins and gems and how easy it was to collaborate using github!

Now I just hope others find the site interesting to use!

I built a blog aggregator - waywework.it

over 3 years ago | Alex Rothenberg: Common Sense Software

I've been spending some time recently putting together a blog aggregator site for some of the folks I work with. Its now up and running at http://waywework.it. I hope this will be an interesting place to share our public community and as one of my colleagues said "this keeps my Google Reader much neater".

Today I'd like to talk about the code running this site which is posted and available on github at http://github.com/alexrothenberg/waywework.

I started thinking I would use an existing aggregator site and just apply my skin but when I did a quick search on github I most of the hard work existed in atom and rss gems and plugins and I wanted to take advantage of the just released Rails 2.2 so I decided to build my own. This turned out to be not too much work. Today I'd like to talk about how I put this together.

First I created my project with some scaffolding for feeds which would have_many posts

class Feed < ActiveRecord::Base
  has_many :posts, :dependent => :delete_all
end

class Post < ActiveRecord::Base
  belongs_to :feed
end



I soon found the atom gem and rss parser built into ruby. Using them was a piece of cake as all I had to do was create a method to call each one in my Feed model

class Feed < ActiveRecord::Base
  def get_posts_from_atom atom_xml
    feed = Atom::Feed.new(atom_xml)
    feed.entries.each { |entry|
      link = entry.links.detect {|l| l.rel == 'alternate'}
      create_post(:contents=>entry.content.value, :url=>link.href, :title=>entry.title, 
                  :published=>entry.published.to_s(:db), :updated=>entry.updated.to_s(:db))
    }
    return !feed.entries.blank?
  end  
  
  def get_posts_from_rss rss_xml
    rss = RSS::Parser.parse(rss_xml, false)
    rss.items.each { |entry|
      create_post(:contents=>entry.description, :url=>entry.link, :title=>entry.title, 
                  :published=>entry.date.to_formatted_s(:db), :updated=>entry.date.to_formatted_s(:db))
    }
    return !rss.items.blank?
  end
end



Of course I had to create the glue wrapping it all together. A rake task to be call on a schedule

namespace :feeds do
  desc "Load the feeds"
  task :populate => :environment do
    feeds = Feed.all
    feeds.each do |feed|
      feed.get_latest
    end
  end
end



and the logic to load the feed, parse it and update the posts.

class Feed < ActiveRecord::Base
  def get_latest
    puts "getting feed for #{name}"
    xml = get_feed
    got_atom_posts = get_posts_from_atom xml
    get_posts_from_rss xml unless got_atom_posts
  end

  def get_feed
    uri = URI.parse(feed_url)
    uri.read
  end
  
  def create_post params
    params.merge!(:feed_id=>id) 
    existing_post = Post.find_by_url(params[:url])
    if existing_post
      existing_post.update_attributes(params)
    else
      Post.create(params) 
    end
  end
end



The next step was to publish an atom feed of my site. Again there was a plugin atom_feed_helperwaiting to help me. I installed the plugin and created a view builder

atom_feed(:url => atom_feed_url) do |feed|
  feed.title("WayWeWork")
  feed.updated(@posts.first.published)

  for post in @posts
    feed.entry(post, :url=>post.url, :published=>post.published, :updated=>post.updated) do |entry|
      entry.title("#{post.feed.author}: #{post.title}")
      entry.content(post.contents, :type => 'html')
    end
  end
end


This was all so easy I had hardly done anything other than glue these plugins together. Now I finished up with a few bells and whistles.

I added a who's talking and archive by date section to my homepage that I called from my controller like this

class PostsController < ApplicationController
  @active_feeds = Feed.by_author
  @activity_by_date = Post.activity_by_date
end



I added security to restrict who can administer feeds

class FeedsController < ApplicationController
  before_filter :authenticate

protected
  def authenticate
     authenticate_or_request_with_http_basic do | user_name, password|
       username = YAML::load_file(File.join(RAILS_ROOT, %w[config password.yml]))['username']
       pwd = YAML::load_file(File.join(RAILS_ROOT, %w[config password.yml]))['password']
      user_name == username && password == pwd
    end
  end
end



For the UI I am somewhat graphically challenged so got some help. For this github was very cool as I could add lessallan as a collaborator and he could check in his changes so they just appeared!

Finally a little work with capistrano (mostly just creating a Capfile) and I could deploy!

Overall I spent a few days and now have a site that does exactly what I want. Where most of the code I wrote is specific to my site and the general purpose plumbing was downloaded. I'm very pleased with the availability of plugins and gems and how easy it was to collaborate using github!

Now I just hope others find the site interesting to use!

Using Ruby for shell scripting

over 3 years ago | Alex Rothenberg: Common Sense Software

Someone came to me the other day with a problem. He had downloaded a directory containing a large number of files with spaces in their filenames and needed to get rid of the spaces so he could load them into the tool he was using. He asked me if I knew of a mass rename tool. I didn't know of such a tool but thought it wouldn't be too hard to write something simple.

First I thought of capturing the output of a find and using a text editor search-and-replace create a whole number of mv old_file new_file_without_spaces commands that could all be run. But then I would have to work with some editor's search-and-replace syntax.

Then I thought of writing a shell script. I started to think about find . -e ... but realized I always get confused about the -e syntax and using sed or awk.

Suddenly I realized I could do this with Ruby. About 10 minutes later I had this simple 3 line script, ran it and was done.


Dir[File.expand_path("#{~/file_with_spaces}/*")].uniq.sort.each do |file|
system "mv '#{file}' '#{file.gsub(' ', '_')}'"
end



Ruby is my new shell scripting language and I don't think I'm ever writing a bash shell script again!

Using Ruby for shell scripting

over 3 years ago | Alex Rothenberg: Common Sense Software

Someone came to me the other day with a problem. He had downloaded a directory containing a large number of files with spaces in their filenames and needed to get rid of the spaces so he could load them into the tool he was using. He asked me if I knew of a mass rename tool. I didn't know of such a tool but thought it wouldn't be too hard to write something simple.

First I thought of capturing the output of a find and using a text editor search-and-replace create a whole number of mv old_file new_file_without_spaces commands that could all be run. But then I would have to work with some editor's search-and-replace syntax.

Then I thought of writing a shell script. I started to think about find . -e ... but realized I always get confused about the -e syntax and using sed or awk.

Suddenly I realized I could do this with Ruby. About 10 minutes later I had this simple 3 line script, ran it and was done.


Dir[File.expand_path("#{~/file_with_spaces}/*")].uniq.sort.each do |file|
system "mv '#{file}' '#{file.gsub(' ', '_')}'"
end



Ruby is my new shell scripting language and I don't think I'm ever writing a bash shell script again!

Undefining a constant in a rspec test

over 3 years ago | Alex Rothenberg: Common Sense Software

I want to share something I just figured out. I had to write some code that used a class from a plugin if it was there and but did not require the plugin. The problem I faced was how to test this.

My method looked like this

class User 
def lookup_additional_person_info
if defined? CommonServices::Person
person = CommonServices::Person.find_by_id(@person_id)
@address = person.address
else
@address = 'not available'
end
end
end

I knew how to define CommonServices::Person to test the positive case but what about the negative? I found out you can undefine a class by sending its parent :remove_constant

  it 'should default additional parameters if CommonServices::Person is not defined' do
defined?(CommonServices::Person).should be_nil
user = User.new(:person_id=123)
user.lookup_additional_person_info
user.address.should == 'not available'
end

it 'should not populate additional parameters if CommonServices::Person is not defined' do
begin
defined?(CommonServices::Person).should be_nil
module CommonServices
class Person
end
end
CommonServices::Person.expects(:find).with(123).returns(person=mock)
person.expects(:address).returns(address_from_service=mock)

user = User.new(:person_id=123)
user.lookup_additional_person_info
user.address.should == address_from_service
ensure
CommonServices.send(:remove_const, :Person)
end
end

Testing with RSpec stories

over 3 years ago | Alex Rothenberg: Common Sense Software

I recently completed a project where we used RSpec stories for our integration testing and wanted to share some of my experiences. Overall RSpec Stories are an incredibly powerful tool and have allowed us to take a huge step towards business analysts writing functional specifications (in the form of plain text stories) then allowing the developers to make them pass. Today I'd like to share how we setup our stories and some of the decisions we made along the way.

The first goal was to run the stories from rake so we created a rake task in lib/tasks/stories.rake to run the stories and fail if any of them fail - this was useful when running this from our cruise server.


task :all do
args = ENV['story_prefix'] || ''
stories_passed = system 'ruby', File.join(RAILS_ROOT, "stories", "run_with_all_steps.rb"), args
raise('story failed') unless stories_passed
end


You can invoke this task as rake stories:all or rake stories:all story_prefix=user_ and the second variant allow you to run just one (or a subset) of the stories which turned out to be useful as they take a long time to run. Now we'll move onto stories/run_with_all_steps.rb which is what actually runs the stories.

First some setup of the environment since we're invoked from the command line


file_pattern = ARGV[0] || "*"
ENV["RAILS_ENV"] = "test"
require File.expand_path(File.dirname(__FILE__) + "/../config/environment")
require 'spec/rails/story_adapter'


Then two helper functions that load all stories in a directory or use the story_pattern passed in. This is something I haven't seen other people do with the RSpec story framework but seemed natural to me as I didn't want to have to maintain a list of stories to run and possibly (or probably) forget to add new ones. Stories also have a concept of steps that go along with stories but I didn't want to have to to manually associate steps with stories so decided to load them all. There were some clashes with different step files using the same phrase to define their steps but when that happened we just changed the phrasing of one or the other. The only trick in here is the sort on the step_files which was required because windows and the mac and linux loaded the files in a different order causing inconsistency across machines.


def all_local_stories(story_pattern = "")
story_path = "#{File.dirname(__FILE__)}/stories"
Dir[File.expand_path("#{story_path}/**/#{story_pattern}*.story")]
end

def all_steps_for_local_stories
steps_path = "#{File.dirname(__FILE__)}/steps"
step_files = Dir[File.expand_path("#{steps_path}/*.rb")].sort
step_files.collect do |file|
require file
file.match(/(\w+)\.rb\Z/).captures[0].underscore.to_sym
end
end


Lastly we invoke the rspec story runner. What's tricky here is that the story runner doesn't run here but registers itself using Kernel.at_exit (which I'd never seen before)


stories = all_local_stories(file_pattern)
stories.each do |file|
with_steps_for(*all_steps_for_local_stories) do
run file, :type => RailsStory
end
end


So up to here we can invoke our stories using rake ... but what are stories and how do they work? Stories are just text files written with the template


Story: Manage Users
As an Administrator
I want to manage the users who have access to my application
So that private stuff isn't seen by the wrong people

Scenario: Revoke access for a user
Given Alex is a registered user
When I delete the user Alex
Then Alex should not be able to access the sensitive content


This is pretty cool ... so how does it work? This is where the steps come in. We define a steps file to match each of the Given/When/Then clauses in our story and define what to do for each one.


require File.dirname(__FILE__) + '/../../../spec/spec_helper'

steps_for(:user_management) do
Given('$user_name is a registered user') do |user_name|
@users ||= []
@users << name =""> user_name})
end

When('I delete the user $user_name') do |user_name|
this_user = @users.detect {|a_user| a_user.name == user_name}
this_user.should_not be_nil
this_user.destroy
end

Then('$user_name should not be able to access the site') do |user_name|
get("/login?username=#{user_name}")
response.response_code.should == 401
end
end


As the rspec story runner parses the story file it looks for a step that matches invokes its block passing a parameter for each $-starting variable in the clause. The way I've written what's above we try to maintain some consistency between the variables referred to in the different clauses. If you refer to the user_name as 'Alex' in the Given clause we store that user away in the @users array and load it from there later on when we want to refer to the same user in the When clause.

We've had pretty good success working with stories in this way and as we move forward are going to continue to experiment with

  • Having our business analysts write most of the story files and the developers write the steps.
  • Thinking of ways to keep the steps files DRY
  • Working with WATIR or Selenium to create stories that drive the browser

Undefining a constant in a rspec test

over 3 years ago | Alex Rothenberg: Common Sense Software

I want to share something I just figured out. I had to write some code that used a class from a plugin if it was there and but did not require the plugin. The problem I faced was how to test this.

My method looked like this

class User 
def lookup_additional_person_info
if defined? CommonServices::Person
person = CommonServices::Person.find_by_id(@person_id)
@address = person.address
else
@address = 'not available'
end
end
end

I knew how to define CommonServices::Person to test the positive case but what about the negative? I found out you can undefine a class by sending its parent :remove_constant

  it 'should default additional parameters if CommonServices::Person is not defined' do
defined?(CommonServices::Person).should be_nil
user = User.new(:person_id=123)
user.lookup_additional_person_info
user.address.should == 'not available'
end

it 'should not populate additional parameters if CommonServices::Person is not defined' do
begin
defined?(CommonServices::Person).should be_nil
module CommonServices
class Person
end
end
CommonServices::Person.expects(:find).with(123).returns(person=mock)
person.expects(:address).returns(address_from_service=mock)

user = User.new(:person_id=123)
user.lookup_additional_person_info
user.address.should == address_from_service
ensure
CommonServices.send(:remove_const, :Person)
end
end

Testing with RSpec stories

over 3 years ago | Alex Rothenberg: Common Sense Software

I recently completed a project where we used RSpec stories for our integration testing and wanted to share some of my experiences. Overall RSpec Stories are an incredibly powerful tool and have allowed us to take a huge step towards business analysts writing functional specifications (in the form of plain text stories) then allowing the developers to make them pass. Today I'd like to share how we setup our stories and some of the decisions we made along the way.

The first goal was to run the stories from rake so we created a rake task in lib/tasks/stories.rake to run the stories and fail if any of them fail - this was useful when running this from our cruise server.


task :all do
args = ENV['story_prefix'] || ''
stories_passed = system 'ruby', File.join(RAILS_ROOT, "stories", "run_with_all_steps.rb"), args
raise('story failed') unless stories_passed
end


You can invoke this task as rake stories:all or rake stories:all story_prefix=user_ and the second variant allow you to run just one (or a subset) of the stories which turned out to be useful as they take a long time to run. Now we'll move onto stories/run_with_all_steps.rb which is what actually runs the stories.

First some setup of the environment since we're invoked from the command line


file_pattern = ARGV[0] || "*"
ENV["RAILS_ENV"] = "test"
require File.expand_path(File.dirname(__FILE__) + "/../config/environment")
require 'spec/rails/story_adapter'


Then two helper functions that load all stories in a directory or use the story_pattern passed in. This is something I haven't seen other people do with the RSpec story framework but seemed natural to me as I didn't want to have to maintain a list of stories to run and possibly (or probably) forget to add new ones. Stories also have a concept of steps that go along with stories but I didn't want to have to to manually associate steps with stories so decided to load them all. There were some clashes with different step files using the same phrase to define their steps but when that happened we just changed the phrasing of one or the other. The only trick in here is the sort on the step_files which was required because windows and the mac and linux loaded the files in a different order causing inconsistency across machines.


def all_local_stories(story_pattern = "")
story_path = "#{File.dirname(__FILE__)}/stories"
Dir[File.expand_path("#{story_path}/**/#{story_pattern}*.story")]
end

def all_steps_for_local_stories
steps_path = "#{File.dirname(__FILE__)}/steps"
step_files = Dir[File.expand_path("#{steps_path}/*.rb")].sort
step_files.collect do |file|
require file
file.match(/(\w+)\.rb\Z/).captures[0].underscore.to_sym
end
end


Lastly we invoke the rspec story runner. What's tricky here is that the story runner doesn't run here but registers itself using Kernel.at_exit (which I'd never seen before)


stories = all_local_stories(file_pattern)
stories.each do |file|
with_steps_for(*all_steps_for_local_stories) do
run file, :type => RailsStory
end
end


So up to here we can invoke our stories using rake ... but what are stories and how do they work? Stories are just text files written with the template


Story: Manage Users
As an Administrator
I want to manage the users who have access to my application
So that private stuff isn't seen by the wrong people

Scenario: Revoke access for a user
Given Alex is a registered user
When I delete the user Alex
Then Alex should not be able to access the sensitive content


This is pretty cool ... so how does it work? This is where the steps come in. We define a steps file to match each of the Given/When/Then clauses in our story and define what to do for each one.


require File.dirname(__FILE__) + '/../../../spec/spec_helper'

steps_for(:user_management) do
Given('$user_name is a registered user') do |user_name|
@users ||= []
@users << name =""> user_name})
end

When('I delete the user $user_name') do |user_name|
this_user = @users.detect {|a_user| a_user.name == user_name}
this_user.should_not be_nil
this_user.destroy
end

Then('$user_name should not be able to access the site') do |user_name|
get("/login?username=#{user_name}")
response.response_code.should == 401
end
end


As the rspec story runner parses the story file it looks for a step that matches invokes its block passing a parameter for each $-starting variable in the clause. The way I've written what's above we try to maintain some consistency between the variables referred to in the different clauses. If you refer to the user_name as 'Alex' in the Given clause we store that user away in the @users array and load it from there later on when we want to refer to the same user in the When clause.

We've had pretty good success working with stories in this way and as we move forward are going to continue to experiment with

  • Having our business analysts write most of the story files and the developers write the steps.
  • Thinking of ways to keep the steps files DRY
  • Working with WATIR or Selenium to create stories that drive the browser

Perhaps outsourcing is harder than offshoring

over 3 years ago | Alex Rothenberg: Common Sense Software

I've been thinking a lot recently about why my current company seems to have so much trouble working with our offshore development partner in India. Most of the people I work with spend their time thinking about how to work with a bunch of people who are halfway around the world with only a few hours when we're all awake at the same time. The way we've approached the problem is to take about 20% of the developers and temporarily move them to the US to work alongside us while trying to use technology to collaborate as best we can with the remaining developers (conference calls, IM, email and occasional visits of the US staff to India). We have managed to overcome some of the distance and I personally know many of the developers by all these channels and have met a large number in person (in the US and India), but problems still remain.

I know we're not the only company to face these problems (in fact a recent blog by Jay Fields Is Distributed Development Viable is what got me thinking on this topic this week). I agree with most of Jay's post but have been wondering If we solved the distributed development problem ... would the problems I see everyday disappear? I have to say they wouldn't and when I ask myself why I've concluded that our problem has less to do with the geographic distribution issues we face (which are certainly formidable!!) and more to do with the basic fact that most of our developers work for a different company than we do. I think outsourcing seems to be as big if not a bigger problem than offshoring.

What do I mean by that...In my experience successfully developing software is largely dependent on having a team of good people who know each other, have worked together before and are comfortable using their critical thinking and expressing their opinions. I imagine that we decided the rate differential wasn't important and flew the entire India based development team to the US to sit next to us tomorrow. The problem we would still have is that the members of my development team would still work for a different company than I do. My company spends a lot of time helping its employees think about their professional development and how to advance their careers. For our developers at our outsourced partner we do none of that (one of the advantages outsourcing is supposed to grant us the advantage of a bigger pool of resources to use but I believe that we don't need "resources" we need "developers" - sorry to dive into my liberal arts education and deconstructionism a little bit :) With outsourcing we give up on thinking of our developers as individuals and spending time and energy (and money) on their professional development and ensuring that they are challenged and want to continue working with us. The effect is that the average tenure of most of these developers is measured in months instead of years. The problem cuts both ways as the developers have an institutional loyalty to their employer and know they will likely not be working with us for the long term so its very hard for many of them to assume the role of trusted team member who makes decisions for the long term.

What's the solution... I don't know. All I know is that the current situation is very difficult and only focusing on shrinking the geographical distance between the US and India does not seem likely to work.

If you made it hear thanks for reading, I needed to get this off my chest. I hope to get back to more technical topics with my next post!

Perhaps outsourcing is harder than offshoring

over 3 years ago | Alex Rothenberg: Common Sense Software

I've been thinking a lot recently about why my current company seems to have so much trouble working with our offshore development partner in India. Most of the people I work with spend their time thinking about how to work with a bunch of people who are halfway around the world with only a few hours when we're all awake at the same time. The way we've approached the problem is to take about 20% of the developers and temporarily move them to the US to work alongside us while trying to use technology to collaborate as best we can with the remaining developers (conference calls, IM, email and occasional visits of the US staff to India). We have managed to overcome some of the distance and I personally know many of the developers by all these channels and have met a large number in person (in the US and India), but problems still remain.

I know we're not the only company to face these problems (in fact a recent blog by Jay Fields Is Distributed Development Viable is what got me thinking on this topic this week). I agree with most of Jay's post but have been wondering If we solved the distributed development problem ... would the problems I see everyday disappear? I have to say they wouldn't and when I ask myself why I've concluded that our problem has less to do with the geographic distribution issues we face (which are certainly formidable!!) and more to do with the basic fact that most of our developers work for a different company than we do. I think outsourcing seems to be as big if not a bigger problem than offshoring.

What do I mean by that...In my experience successfully developing software is largely dependent on having a team of good people who know each other, have worked together before and are comfortable using their critical thinking and expressing their opinions. I imagine that we decided the rate differential wasn't important and flew the entire India based development team to the US to sit next to us tomorrow. The problem we would still have is that the members of my development team would still work for a different company than I do. My company spends a lot of time helping its employees think about their professional development and how to advance their careers. For our developers at our outsourced partner we do none of that (one of the advantages outsourcing is supposed to grant us the advantage of a bigger pool of resources to use but I believe that we don't need "resources" we need "developers" - sorry to dive into my liberal arts education and deconstructionism a little bit :) With outsourcing we give up on thinking of our developers as individuals and spending time and energy (and money) on their professional development and ensuring that they are challenged and want to continue working with us. The effect is that the average tenure of most of these developers is measured in months instead of years. The problem cuts both ways as the developers have an institutional loyalty to their employer and know they will likely not be working with us for the long term so its very hard for many of them to assume the role of trusted team member who makes decisions for the long term.

What's the solution... I don't know. All I know is that the current situation is very difficult and only focusing on shrinking the geographical distance between the US and India does not seem likely to work.

If you made it hear thanks for reading, I needed to get this off my chest. I hope to get back to more technical topics with my next post!

Working with Excel data without working with Excel

over 3 years ago | Alex Rothenberg: Common Sense Software

I am currently working on an application to replace manual MS Excel files that get emailed around and while the business users are looking forward to an interactive web app they are also wedded to their excel files. One of the key requirements we need to meet is allowing them to move their data back and forth.

In the past I've worked with COM and Java libraries to manipulate the files in their native formats and both were painful experiences I did not want to relive. I'm going to talk about how we solved the problem without the need to work with any Office libraries.


Loading Excel data into our Application
The approach we took was to let the users extract the data themselves with a simple copy-and-paste. The user would copy the cells of the excel file and then paste that tab delimited plain text into an HTML form.

Let's say we're trying to import a list of people with a first and last name. The Excel sheet would look like this


We start with a form with a big text area and submit button and an area to show any errors we may encounter while parsing

<% form_tag do -%>
Paste your Excel data : <%= text_area_tag :data, @data %>
Click to validate the data <%= submit_tag 'Validate' %>

<ul>
<% @errors.each do |error| %>
<li><span ><%=error%></span></li>

<% end unless @errors.blank? -%>
</ul>
Then the controller action which let's the model parse the text and only saves if every person model is valid.
class PeopleController <  ApplicationController
def import
@errors = []
people = Person.all_from_tab_delimited_text(params[:data])

people.each do |person|
person.errors.each { |error| @errors << "#{error} on #{person.inspect}"} unless person.valid?
end
end
end
Now we move onto the Person model where the parsing actually takes place
class
class Person < ActiveRecord::Base
def Person.all_from_tab_delimited_text(data)
people = []
CSV::Reader.parse(@data, "\t") do |row|
person = Person.new(:first_name"> row[0], :last_name => row[1])
people << person
end
people
end
end
We now have a working allbeit simple implementation that even does some simple error checking to give the user feedback if their data cannot be imported. Of course the example here is simple and the real system has additional features and complexity that I didn't show here. Some of these are
  • The import into a 2-step operation where we first ask the user to click 'Validate' then return a table showing the data as we parsed it. This allows the user to review and verify it was parsed correctly (i.e. the columns were in the correct order). Only on this second page do we provide an 'import' button that actually persists the new data.
  • The data can not only create new people but also update existing ones. The example above with just first and last name is contrived so this wouldn't make sense but imagine an example where there are many other columns like address, phone number, etc. Our model code is a bit more complex in that it first tries to find a person with the same first and last names. If it succeeds it will then update that person and only creates a new one if there is no match.
  • Some of the additional columns in the real system refer to other models in our system. To deal with these our model code must find other kinds of model objects and pass those into the Person.new. Ensuring the proper errors are sent back to the user in this case is fairly tricky as they may not always show up as part of the ActiveRecord validations.
Downloading data from our app into Excel
Moving data in this direction was even easier as we just relied on the fact that Excel will open csv files as long as our Rails app returns an Excel MIME type. We add a line to register the mime type with the csv extension
# config/initializers/mime_types.rb
Mime::Type.register 'application/vnd.ms-excel', :csv
Then we had to do for this was add a respond_to.xsl in the appropriate places of our controller
class PeopleController < ApplicationController
def index
people = Person.all
respond_to do |format|
format.html # index.html.erb
format.csv do
render :text => @people.collect { |person| person.to_csv}.join("\n")
end
end
end
and then add a to_csv to the model
class Person < ActiveRecord::Base
def to_csv
"#{first_name}\t#{last_name}"
end
end

This only works for very simple Excel data on a single worksheet but when that's true it makes your job of working with that data so much simpler!

Working with Excel data without working with Excel

over 3 years ago | Alex Rothenberg: Common Sense Software

I am currently working on an application to replace manual MS Excel files that get emailed around and while the business users are looking forward to an interactive web app they are also wedded to their excel files. One of the key requirements we need to meet is allowing them to move their data back and forth.

In the past I've worked with COM and Java libraries to manipulate the files in their native formats and both were painful experiences I did not want to relive. I'm going to talk about how we solved the problem without the need to work with any Office libraries.


Loading Excel data into our Application
The approach we took was to let the users extract the data themselves with a simple copy-and-paste. The user would copy the cells of the excel file and then paste that tab delimited plain text into an HTML form.

Let's say we're trying to import a list of people with a first and last name. The Excel sheet would look like this


We start with a form with a big text area and submit button and an area to show any errors we may encounter while parsing

<% form_tag do -%>
Paste your Excel data : <%= text_area_tag :data, @data %>
Click to validate the data <%= submit_tag 'Validate' %>

<ul>
<% @errors.each do |error| %>
<li><span ><%=error%></span></li>

<% end unless @errors.blank? -%>
</ul>
Then the controller action which let's the model parse the text and only saves if every person model is valid.
class PeopleController <  ApplicationController
def import
@errors = []
people = Person.all_from_tab_delimited_text(params[:data])

people.each do |person|
person.errors.each { |error| @errors << "#{error} on #{person.inspect}"} unless person.valid?
end
end
end
Now we move onto the Person model where the parsing actually takes place
class
class Person < ActiveRecord::Base
def Person.all_from_tab_delimited_text(data)
people = []
CSV::Reader.parse(@data, "\t") do |row|
person = Person.new(:first_name"> row[0], :last_name => row[1])
people << person
end
people
end
end
We now have a working allbeit simple implementation that even does some simple error checking to give the user feedback if their data cannot be imported. Of course the example here is simple and the real system has additional features and complexity that I didn't show here. Some of these are
  • The import into a 2-step operation where we first ask the user to click 'Validate' then return a table showing the data as we parsed it. This allows the user to review and verify it was parsed correctly (i.e. the columns were in the correct order). Only on this second page do we provide an 'import' button that actually persists the new data.
  • The data can not only create new people but also update existing ones. The example above with just first and last name is contrived so this wouldn't make sense but imagine an example where there are many other columns like address, phone number, etc. Our model code is a bit more complex in that it first tries to find a person with the same first and last names. If it succeeds it will then update that person and only creates a new one if there is no match.
  • Some of the additional columns in the real system refer to other models in our system. To deal with these our model code must find other kinds of model objects and pass those into the Person.new. Ensuring the proper errors are sent back to the user in this case is fairly tricky as they may not always show up as part of the ActiveRecord validations.
Downloading data from our app into Excel
Moving data in this direction was even easier as we just relied on the fact that Excel will open csv files as long as our Rails app returns an Excel MIME type. We add a line to register the mime type with the csv extension
# config/initializers/mime_types.rb
Mime::Type.register 'application/vnd.ms-excel', :csv
Then we had to do for this was add a respond_to.xsl in the appropriate places of our controller
class PeopleController < ApplicationController
def index
people = Person.all
respond_to do |format|
format.html # index.html.erb
format.csv do
render :text => @people.collect { |person| person.to_csv}.join("\n")
end
end
end
and then add a to_csv to the model
class Person < ActiveRecord::Base
def to_csv
"#{first_name}\t#{last_name}"
end
end

This only works for very simple Excel data on a single worksheet but when that's true it makes your job of working with that data so much simpler!

How to unit test a Java class with static initializers

over 3 years ago | Alex Rothenberg: Common Sense Software

We have a large Java codebase that we're trying to put under test. Since this class was not designed for testability we often run into code that cannot be tested as is. Most often this involves code that assumes it will always be run inside our J2EE Application Server because it depends on classes provided by the server.

I was recently working with someone on this Java class

class Person {
static {
Logger logger = new ContainerLogger();
}

public String doSomething() {
...
logger.info("I did something");
...
}
}

We couldn't create a unit test for the doSomething method because the ContainerLogger class was provided by our application server and couldn't be used outside the container. We refactored to come up with this solution.

class MyClass {
public static Logger _logger;
public Logger logger() {
if (_logger == null) {
logger = new ContainerLogger();
}
return _logger;
}

public String doSomething() {
...
logger().info("I did something");
...
}
}

We refactored our code to access the logger using the accessor method logger() then we also made the logger variable itself public at the same time. Why did we do those two seemingly contradictory things? The accessor method implements the singleton pattern preserving the semantics of the static initializer and the public access to the instance variable _logger allows us to replace the implementation with a mock in our test.

public class TestMyClass extends TestCase {
public void testSomething() {
MyClass._logger = new MockLogger();
assert(MyClass.doSomething(), "expected return");
}
}

I've seen too much legacy Java code that assumes it will always run inside a container but with testing we need to change our mindsets because when running a test we will be outside the container. All dependencies on the container (and in most applications I've seen we don't need that many!) should be encapsulated and able to be mocked through dependency injection. If you do this you'll end up with simpler code that better follows the Single Responsibility Principle

How to unit test a Java class with static initializers

over 3 years ago | Alex Rothenberg: Common Sense Software

We have a large Java codebase that we're trying to put under test. Since this class was not designed for testability we often run into code that cannot be tested as is. Most often this involves code that assumes it will always be run inside our J2EE Application Server because it depends on classes provided by the server.

I was recently working with someone on this Java class

class Person {
static {
Logger logger = new ContainerLogger();
}

public String doSomething() {
...
logger.info("I did something");
...
}
}

We couldn't create a unit test for the doSomething method because the ContainerLogger class was provided by our application server and couldn't be used outside the container. We refactored to come up with this solution.

class MyClass {
public static Logger _logger;
public Logger logger() {
if (_logger == null) {
logger = new ContainerLogger();
}
return _logger;
}

public String doSomething() {
...
logger().info("I did something");
...
}
}

We refactored our code to access the logger using the accessor method logger() then we also made the logger variable itself public at the same time. Why did we do those two seemingly contradictory things? The accessor method implements the singleton pattern preserving the semantics of the static initializer and the public access to the instance variable _logger allows us to replace the implementation with a mock in our test.

public class TestMyClass extends TestCase {
public void testSomething() {
MyClass._logger = new MockLogger();
assert(MyClass.doSomething(), "expected return");
}
}

I've seen too much legacy Java code that assumes it will always run inside a container but with testing we need to change our mindsets because when running a test we will be outside the container. All dependencies on the container (and in most applications I've seen we don't need that many!) should be encapsulated and able to be mocked through dependency injection. If you do this you'll end up with simpler code that better follows the Single Responsibility Principle

Some things are inherently complicated and slow ... put them in the right place

over 3 years ago | Alex Rothenberg: Common Sense Software

One of the first lessons I learned when I started working as a software developer professionally was that you don't always have to make an action fast, sometimes its just enough to make it seem fast to a user. This lesson was learned in the 1990s in the context of a Windows application. What we did was quickly draw part of the screen (or a splash image) immediately then do the time consuming work in the background so that by the time the user was ready to interact with the application we'd be ready for them.

Recently I was reminded of this lesson on a website I'm working on. We have a complicated report to show on the user's homepage. My first implementation involved generating the report in real-time when the page is loaded but this was very slooooow. Some profiling and analysis let us make it somewhat faster but not fast enough.

I was stumped until I remembered my old lesson. If it wasn't possible to generate the report quickly, perhaps we could do it sometime when the user wouldn't mind.

Luckily this is a Rails application and ActiveRecord Callbacks make it easy to do this. I could pregenerate the report and update a portion of it each time something is saved. Users expect a save to take some time and don't do it that often. Then generating the homepage becomes just a simple matter of displaying the existing rows.

First I setup my models to call into the report generator every time they change

class Person < ActiveRecord::Base
  after_destroy {|person| ReportGenerator::Person.destroyed(person)}
  after_create  {|person| ReportGenerator::Person.created(person)  }
  after_update  {|person| ReportGenerator::Person.updated(person)  }
end



Then I implement the complicated (and slow) logic to pre-compute the rows in the report in a library class

module ReportGenerator
  class Executive
    def destroyed(executive)
      #figure out which rows to delete from the report and persist with the Report model
    end
    def created(executive)
      #figure out which rows to add to the report and persist with the Report model
    end
    def updated(executive)
      #figure out which rows to update in the report and persist with the Report model
    end
  end
  #Similar classes corresponding to the other models that trigger recalculations would go here
end



Finally I can show the report on the homepage with some boring (and fast) Rails code

Model:

class Report < ActiveRecord::Base
end



Controller:

class ReportsController < ApplicationController
  def index
    @report =" ReportRows.find_by_user(current_user)">
  end
end



and the View:

<% for row in @report %>
  Display the row...
<% end %>



The interesting insight for me is that when optimizing there are sometimes hard problems that can't be solved. Its important not to lose sight of the goal you're aiming at (a satisfying user experience) and that sometimes involves spending the computational time somewhere where the user won't mind.

Some things are inherently complicated and slow ... put them in the right place

over 3 years ago | Alex Rothenberg: Common Sense Software

One of the first lessons I learned when I started working as a software developer professionally was that you don't always have to make an action fast, sometimes its just enough to make it seem fast to a user. This lesson was learned in the 1990s in the context of a Windows application. What we did was quickly draw part of the screen (or a splash image) immediately then do the time consuming work in the background so that by the time the user was ready to interact with the application we'd be ready for them.

Recently I was reminded of this lesson on a website I'm working on. We have a complicated report to show on the user's homepage. My first implementation involved generating the report in real-time when the page is loaded but this was very slooooow. Some profiling and analysis let us make it somewhat faster but not fast enough.

I was stumped until I remembered my old lesson. If it wasn't possible to generate the report quickly, perhaps we could do it sometime when the user wouldn't mind.

Luckily this is a Rails application and ActiveRecord Callbacks make it easy to do this. I could pregenerate the report and update a portion of it each time something is saved. Users expect a save to take some time and don't do it that often. Then generating the homepage becomes just a simple matter of displaying the existing rows.

First I setup my models to call into the report generator every time they change

class Person < ActiveRecord::Base
  after_destroy {|person| ReportGenerator::Person.destroyed(person)}
  after_create  {|person| ReportGenerator::Person.created(person)  }
  after_update  {|person| ReportGenerator::Person.updated(person)  }
end



Then I implement the complicated (and slow) logic to pre-compute the rows in the report in a library class

module ReportGenerator
  class Executive
    def destroyed(executive)
      #figure out which rows to delete from the report and persist with the Report model
    end
    def created(executive)
      #figure out which rows to add to the report and persist with the Report model
    end
    def updated(executive)
      #figure out which rows to update in the report and persist with the Report model
    end
  end
  #Similar classes corresponding to the other models that trigger recalculations would go here
end



Finally I can show the report on the homepage with some boring (and fast) Rails code

Model:

class Report < ActiveRecord::Base
end



Controller:

class ReportsController < ApplicationController
  def index
    @report =" ReportRows.find_by_user(current_user)">
  end
end



and the View:

<% for row in @report %>
  Display the row...
<% end %>



The interesting insight for me is that when optimizing there are sometimes hard problems that can't be solved. Its important not to lose sight of the goal you're aiming at (a satisfying user experience) and that sometimes involves spending the computational time somewhere where the user won't mind.