opensoul.org - Code

Outsourcing vs. Offshoring

2008-12-21T19:32:00Z

My experience is that most people think of “outsourcing” and “offshoring” as synonyms. We talk about the general category of outsourcing with slight repugnance, acknowledging that it is occasionally useful for more mundane and unskilled tasks, but it simply won’t work for the more creative tasks that require an American to do it well. (Maybe nobody blatantly says it requires an American, but we know what you’re implying.)

I don’t intend to address the “Americans are better at creative work” myth, but I will say this: my hypothesis is that any perceived difference in the quality of outsourced work has more to do with the size of the team, access to the stakeholders and end-user, and the ownership and responsibility felt by the people doing the work than it does their nationality, all factors which can also affect an internal team. But I have no experience with offshoring, so I can’t speak to that.

Instead, I want to clarify that offshoring is but one category of outsourcing.

You may not call it “outsourcing”, but you do it all the time

“Outsourcing” simply means paying someone else do work that you could do, usually because they can do it cheaper, better or faster, or any combination of those. We all outsource all of the time. As cofounder of Keepers Household, Inc., I could have our maintenance staff–a position currently held by yours truely–take the time to figure out why the washing machine sometimes doesn’t spin, but not only do I have no interest in this task, I can pay someone else that can diagnose the problem quicker and probably fix it better than I can. I’m exchanging a little money for a working washing machine and a few hours back to do what I’m good at and enjoy¹.

In nearly three years of doing web application development at Collective Idea, I can only think of one or two small projects where we didn’t “outsource” parts of the project. On every other project, we’ve worked in conjunction with one or more companies or individuals on various aspects.

It’s not just about getting it cheaper

Design, for example, is an area that we often outsource. We believe strongly in the importance of good interface design, and doing it as early as possible in the project. We can do design ourselves (or at least pretend to), but it is extremely inefficient for us. It is not a core competency, nor is it an area where we have a real competitive advantage.

So we have two options: hire someone that is good and efficient at design, or outsource it. Most companies choose to hire someone, assuming that they will get a better value and the direct access to the designer will yield quicker and higher quality results.

Instead, we outsource it to one of the designers that we know, and not only is the result much better than we could have managed, it is much cheaper. The result is better because we get to work with better designers—people that we couldn’t afford to employ—and they bring with them the experience of all the other projects that they’ve worked on, successes and failures. While the may bill us at a much higher rate than we could have done it ourselves, in the end, it’s still a better value because they make fewer mistakes and are more efficient.

But ultimately, it’s not about getting design done cheaper, it’s about getting it done better. Cheaper just happens to be a nice side effect. We outsource design because we can’t afford to not do it well, so we want the most creative people we can find.

Not just the mundane

On one end of the spectrum of work is the repetitive, dull tasks that anybody can do. And on the other end is the highly creative or skilled work that only a handful of people can do. I disagree with the notion that only the mundane and uncreative tasks are the candidates for outsourcing. The closer the work is to either edge of that spectrum, the more it could benefit from outsourcing.

Besides design, we’ve also outsourced things that require very specialized knowledge or skills. Occasionally we have interest and can afford to pursue those areas ourselves, but usually it’s more cost-effective and productive to bring in someone that already has experience, for the same reasons we outsource design.

This post is partly inspired by Outsourcing Killed By Django And Ruby On Rails, which argues that the Django and Rails web frameworks have allowed us to compete with the big guns because the frameworks eliminate the 80% of mundane tasks that would otherwise be offshored, allowing us to focus on the creative aspects.

While I agree with their argument that these frameworks eliminate the mundane tasks and allow us to be more efficient, I would argue that more efficiency comes from other factors which have more to do with the culture around these frameworks, such as smaller team sizes, open communication and releasing early which forces you to focus on core features needed by existing users instead of what potential users might want. But I digress…

More importantly, I disagree with their premise that these frameworks have killed outsourcing. From my perspective, they have done exactly the opposite, and it is the ability of small specialized teams to outsource that gives them an even greater edge on the corporate competition.

If frameworks like Django and Rails are killing anything, I would hope that it is the culture of corporate software manufacturing, or the idea that we can just put more cheap labor on the software assembly line and get better results.

I’m writing this right now when I should be fixing the washing machine, but I’ve promised my cofounder/business partner that I’ll call someone to fix it.

Graticule and MapQuest?

2008-12-13T16:15:00Z

MapQuest has sent out several emails about their current geocoding API being discontinued. Here is the latest:

Dear MapQuest Platform Customer,
At the start of the New Year, MapQuest will be retiring the MapQuest OpenAPI product, having launched the more feature rich MapQuest Platform: Free Edition product. Since the MapQuest OpenAPI does not use the same backend as our newer APIs, nor does it provide the breadth in functionality, we want to provide you with a better free experience. Don’t wait – make the switch today!

If your application is currently being powered by the MapQuest OpenAPI product, you will need to migrate to one of 6 APIs available in the MapQuest Platform: Free Edition product prior to January 31, 2009.

Our MapQuest Platform: Free Edition product offers more flexibility and ease of development along with providing developer choice with six APIs:

SERVER SIDE APIs Java C++ .NET

CLIENT SIDE APIs JavaScript AS3 (ActionScript 3: Flash, Flex, AIR) FUJAX (Write JavaScript, output Flash)
Our MapQuest Platform: Free Edition product includes many additional features including:

COLLECTIONS: Support for multiple and remote collections (KML and GeoRSS); allowing easier handling of shape collections.

ADVANCED SHAPE OVERLAYS: Build applications that allow users to create and interact with a variety of overlays on maps, including custom lines, polygons, rectangles, and ellipses.

CUSTOM TILE LAYER SUPPORT.

Add REAL-TIME TRAFFIC to your map.

GLOBE VIEW: http://globe.mapquest.com.

AERIAL IMAGERY and HYBRID VIEWS.

SMART ROLLOVERS: Rollover windows that adapt their size and positioning based on the content placed in the window.

ADVANCED MAP MARKERS: With the MapQuest “declutter mode,” automatically move collided markers to positions on the map with a leader link pointing to their original location.

The MapQuest OpenAPI product servers will go offline on Saturday, January 31st, 2009. Please plan on migrating your application before this date or applications based on the MapQuest OpenAPI product will stop working.

You can find documentation and downloads for the MapQuest Platform: Free Edition product on our Developer Network: http://developer.mapquest.com.

Additional information can be found on: http://platform.mapquest.com & http://devblog.mapquest.com.

Thank you,
MapQuest, Inc.

Graticule, a ruby wrapper around many popular geocoding APIs, uses the old MapQuest API. I don’t use it, and I don’t know if anyone else does, so I’m leaving it to you, lazyweb: If you use the MapQuest API, or want to in the future, fork graticule on GitHub and update the MapQuest wrapper.

Testing views in RSpec with namespaced elements

2008-12-04T23:33:00Z

We’ve been working on a Facebook app (yeah, I know, I’m sorry) and have had all kinds of fun challenges. We wanted to test FBML tags in the view, but the problem is that FBML uses namespaced elements.

It turns out, CSS selectors don’t work with namespaced elements. There is a CSS3 namespace module, but Rails’ #assert_select and RSpec’s #have_tag (built on #assert_select) don’t support them. We were going to have to turn to XPath, but #have_tag only supports CSS selectors.

But Hpricot supports XPath. A little searching revealed the rspec_hpricot_matchers plugin, which replaces have_tag with an Hpricot-backed version. The only problem is that the new version isn’t backwards compatible. So we forked it and renamed #have_tag to #match_element.

Usage

It’s pretty straight forward. Install the rspec_hpricot_matchers plugin, and include the module in the RSpec config in spec_helper.rb.

Spec::Runner.configure do |config|
  config.include RspecHpricotMatchers
end

Our view had something similar to:

<fb:multi-friend-input prefill_ids="<%= @friends.join(', ') %>" />

And in the view spec, we use #match_element with an XPath expression.

describe "things/edit.fbml.erb" do
  it "should prefill selected friends" do
    render "things/edit.fbml.erb"
    response.should match_element("//fb:multi-friend-input[@prefill_ids='333,444,555']")
  end
end

awesome_nested_set: making nested sets cool

2008-11-17T06:32:00Z

Yes, I’m making the assertion that preordered tree traversal is now cool. And I don’t mean just “pocket protector” cool, because it’s always been that, but now it’s “show your friends” cool.

For those that have no idea what I’m talking about, and don’t really care, but still want to be cool, skip to the next section. For all three of you that want to understand all the gory details, check out this MySQL DevZone article on managing hierarchical data.

What are you talking about?

I’m talking about putting hierarchical data into a relational database, and a plugin to make that easier. There are lots of reasons for trying to do this: organizational structures, genealogies, taxonomies, nested pages of a website, etc. It’s kinda like putting a square peg into a round hole, except that the square peg is made out of Play-Doh, so we can force it through the hole anyway, and we just have a little extra mess to deal with.

There were several Active Record plugins out there that tried to clean up the mess, but they were either buggy or incomplete.

We created awesome_nested_set to try to remedy that.

What makes this so awesome?

There’s a lot of things that makes this awesome, but my personal favorite is that awesome_nested_set makes use of Rails 2.1’s named_scope features¹, so most of the nested set methods return a scope that works as a finder. You can call find methods on it or access other named scopes.

class Department < ActiveRecord::Base
  acts_as_nested_set
  named_scope :in_need_of_review, :lambda => {{
    :conditions => {:reviewed_at > 1.year.ago
  }}
end

chancellor = Department.create(:name => 'Chancellor')
aa = Department.create(:name => 'Academic Affairs').move_to_child_of(chancellor)
Department.create(:name => 'Admissions').move_to_child_of aa
Department.create(:name => 'Student Services',
  :reviewed_at => 3.months.ago).move_to_child_of aa

chancellor.descendants.in_need_of_review
rogue = chancellor.descendants.all(:conditions => 'manager_id IS NULL')

There’s lots more info in the README on GitHub, so check it out. Let us know if you have any suggestions or feedback.

It also backports named_scope for those still on Rails 2.0

Capistrano 2.5 and older versions of git

2008-11-13T16:43:00Z

Git support in Capistrano 2.5 got a little lovin’, but as a result, it may have broke your deploys. By default, Capistrano now uses the -q flag to tell git to not be so chatty. But older versions of git (1.4.x and maybe some early 1.5.x versions) don’t support the -q flag. If upgrading git isn’t an option, the solution is simple. Just tell Capistrano to take the muzzle off of git:

set :scm_verbose, true

Rails Training: January 20–23 in San Antonio, TX

2008-11-12T17:32:00Z

We’ve been providing highly-praised Ruby on Rails training all over the world, and now we’re bringing it to San Antonio, Texas. We will be offering hands-on Ruby on Rails training in downtown San Antonio on January 20-23, 2009.

I’ve never been to San Antonio, so I’m really exited for this class. Any recommendations for things to check out?

Making RSpec concise

2008-11-10T03:07:00Z

A common criticism of RSpec is that it is very verbose. I don’t necessarily agree (or care), but I thought it would be fun to see how concise I could make my specs.

Here are some simple specs from a client project:

describe Company do
  before do
    @company = Company.new
  end

  it "should have many classifications" do
    @company.should have_many(:classifications)
  end

  it "should have many industries through companies" do
    @company.should have_many(:industries, :through => :classifications)
  end

  it "should have many locations" do
    @company.should have_many(:locations)
  end

  it "should have many leads" do
    @company.should have_many(:leads)
  end

  it "should have many jobs" do
    @company.should have_many(:jobs)
  end

  it "should have many notes" do
    @company.should have_many(:notes)
  end

  it "should have many phones ordered by phone type position" do
    @company.should have_many(:phones, :as => :phonable,
      :include => :phone_type, :order => 'phone_types.position')
  end

  it "should have many events" do
    @company.should have_many(:events)
  end

  it "should have many interests" do
    @company.should have_many(:interests)
  end

  it "should belong_to an exchange" do
    @company.should belong_to(:exchange)
  end

  it 'should belong_to a primary_contact' do
    @company.should belong_to(:primary_contact, :class_name => 'Job')
  end

  it 'should have many articles' do
    @company.should have_many(:articles,
      :order => 'articles.date DESC, articles.created_at DESC')
  end
end

These specs check the declared associations on our company model using some custom matchers. They are not very complicated, but are somewhat repetitive. Each example has a description that is basically a duplication of the implementation.

Step 1: remove the description

For a while now, RSpec has had the ability for matchers to be self describing. If you don’t pass a block to #it, it uses the description provided by the matcher.

it do
  @company.should have_many(:jobs)
end

When that spec is run, it gives the output “should have a has_many association called :jobs”. Depending on what you’re speccing, the built in description isn’t always clear, but in this case it’s great.

See #simple_matcher if you want to create custom matchers with useful error messages.

Step 2: remove the subject

So the duplication within each example is gone, but if you look at the full spec above, each example calls @company.should. Accessing an instance variable isn’t what I would consider “duplication”, but thanks to a nifty new feature added to RSpec today, it’s now unnecessary noise. We can simply call #should within our example, and it will use a new instance of the described type as the “subject”.

describe Company do
  it do
    should have_many(:jobs)
  end
end

You can customize the subject if you don’t simply want a new instance.

describe Company, 'validations' do
  subject { Company.new(valid_company_attributes) }

  it do
    should be_valid
  end
end

Note: As David Chelimsky points out in the comments, this is not released yet and is subject to change.

Step 3: One-liner

Lastly, we can use the one line block:

describe Company do
  it { should have_many(:classifications) }
  it { should have_many(:events) }
  it { should have_many(:interests) }
  it { should have_many(:jobs) }
  it { should have_many(:leads) }
  it { should have_many(:locations) }
  it { should have_many(:notes) }
  it { should have_many(:articles, :order => 'articles.date DESC, articles.created_at DESC') }
  it { should have_many(:industries, :through => :classifications) }
  it { should have_many(:phones, :as => :phonable, :include => :phone_type, :order => 'phone_types.position') }
  it { should belong_to(:exchange) }
  it { should belong_to(:primary_contact, :class_name => 'Job') }
end

That’s pretty sexy. I wasn’t able to do this with all of the specs in my app, but it worked with quite a few of them.

It's a search party!

2008-11-08T20:03:00Z

While chatting with Dr. John Nunemaker at RubyConf, I realized that I have several problems. Ignoring the many character flaws that are beyond the scope of this post, my problems are:

I tag things I find useful on Delicious. But I rarely look back to delicious because it’s just easier to search Google, resorting to Delicious if I can’t find something that I remember tagging.
It’s easier to search Google because not everything that I find useful is in Delicious. I don’t want to have to think about where useful things are, I just want to search for them.

I want a search engine that prioritizes things that I’ve found useful in the past. Ideally, google would let me tag things and take that into account when calculating page rank. But, in the mean time…

Let’s have a search party!

I threw together a simple search interface that pulls in results from Google, Delicious, and just for fun, GitHub. It shows primarily Google results, but then in the sidebar, it shows results from Delicious, with my bookmarks highlighted at the top, and also results from GitHub. Check it out.

And if you use Firefox, you can add it as your search provider:

It is an extremely simple app that has 2 pages and uses JavaScript to read in JSON from all of the services. There is one Ruby class that does screen-scraping since Delicious doesn’t provide an API to their full search.

I first implemented it in Merb, but upon realizing how simplistic it is, I switched it over to Sinatra. And I used jQuery to do the JSON and other JavaScripty goodness. Thanks to Mark Van Holstyn for helping implement it.

The code is available on GitHub, so check it out, fork it, and make it more awesome.

I will be making a few other posts about specific things I learned when building this, including deploying sinatra apps and using jQuery to do JSON with Merb and Sinatra.

Ajax and Request Forgery Protection

2008-10-24T04:52:00Z

Rails 2.1 added protection for cross-site request forgery by embedding a session-based token in generated forms. Rails will not process a POSTed request without the token. For the most part, this protection is transparent. But occasionally, an Ajax request request gets left out in the cold without a token.

If your Ajax request is tied to a form on the page, then all is good because the form already has the authenticity token in it. It only happens when your Ajax request is not tied to a form but makes a POST request, which is a rare but occasionally useful.

So how do we let those Ajax requests in on the fun? We came up with the ingenious idea of just embedding the authenticity token in a meta tag on every page, which can then be used in the Javascript.

<meta name="authenticity-token" id="authenticity-token" content="<%= form_authenticity_token %>" />

The authenticity token is unique for each visitor, and already included in other parts of the page, so this doesn’t defeat the purpose of the request forgery protection.

We usually add a couple convenience methods for accessing the token in Javascript.

var Application = {
  authenticityToken: function() {
    return $('authenticity-token').content;
  },

  authenticityTokenParameter: function(){
   return 'authenticity_token=' + encodeURIComponent(Application.authenticityToken());
  }
}

Now, we have easy access to it whenever we need it.

new Ajax.Request(url, {
  parameters: Application.authenticityTokenParameter()
});

Push Upstream

2008-10-22T04:18:00Z

Scenario 1: You’re half way through a really productive day on a wicked new feature for an app and everything is going smoothly. Then, from out of nowhere…SMACK! A nasty bug in a library you depend on splatters right in your face. “Seriously, nobody has come across this before?” you mumble.

Scenario 2: You’re using some fancy library and you think to yourself, “Geez, wouldn’t it be sweet if it did <insert fancy feature here> for you?”

The temptation, especially with Ruby, is to solve both of these problems in your app’s code base by modifying your copy of the library. This may seem like the path of least resistance and the quickest way to move forward in the short term, but in the long term all you are doing is delaying the cost of properly solving that problem. You are incurring debt, which over time compounds and will end up costing you more to fix.

First, when you’re making the change just for yourself, you’re less likely to solve the problem properly or do the appropriate testing. You’ll do some hack job for the problem you have, without considering how it might affect other scenarios.

Second, if you found a bug or have a desired feature, chances are that someone else has or will run into the same situation. Fixing it locally doesn’t help everyone else.

Lastly, fixing the bug or adding a new feature directly in your app makes it nearly impossible to upgrade the library. I spent over 4 hours this week trying to upgrade an open source app to the latest version of Ruby on Rails, and still have over 100 failing tests. The biggest problem isn’t Rails itself, but all the plugins that also need upgraded, many of which had been patched in some form.

At Collective Idea, we use git submodules for as many external libraries as possible. While submodules have their own issues, and probably aren’t a recommended approach, the positive side-effect is that it strongly discourages us from patching libraries in place. If we do need to make a change, each library is already its own git repository, so we can push our changes and send a pull request to the original maintainer.

If you need to modify an open source library, don’t just do it in place. Push your changes upstream. Irresponsible patching hurts us all.

Money with precision

2008-10-18T20:26:00Z

I’ve been working on a project that needs to store mileage reimbursement rates to the nearest tenth of a cent. We were using the money gem, which stores money amounts in cents, so it looked like it was going to be a pain.

But without too much suffering, I modified the money gem to take a precision (in powers of 10), which defaults to 2. It can now store amounts in any precision.

>> amount = 20.to_money + 0.505.to_money
=> #<Money @precision=3, @currency="USD", @cents=20505>
>> amount.to_s
=> "20.505"
>> amount.format
=> "$20.51"

You can also store amounts in negative precisions, like millions:

>> amount = Money.new(50, 'USD', -6)
>> amount.to_s
=> "50"
>> amount.format
=> "$50000000.00"

Check out our fork of the money gem on github. There are lots of other goodies in there.

Behavior Driven Development with Cucumber

2008-10-14T04:14:00Z

The Great Lakes Ruby Bash was a good time. There were several quality presentations, including Jim Weirich’s talk “Playing it Safe – How to write library friendly code in Ruby”, Larry Karnowski’s talk “Usability on Rails”, and Brandon Dimcheff’s “Metawhat? A look into the mysterious metaclass”.

I presented a talk titled “Behavior Driven Development with Cucumber”. Despite the fact that half of the audience didn’t know what the hell I was talking about, I think it went well. I uploaded the slides from my talk to slideshare if you’re interested. I’m not sure that they’ll really be very helpful, but I may try to record audio to go with them at some point.

As far as I know, no audio or video was captured at the conference.

acts_as_audited in development

2008-10-13T02:10:00Z

Thanks to metatribe, a huge annoyance in acts_as_audited is now fixed: it works in development mode. Due to it’s dependance on Rails’ cache sweepers, which are only enabled when caching is, acts_as_audited didn’t work in development mode.

metatribe has a crafty solution, which took me a few minutes to believe it would even work. So pull the latest version and happy auditing.

The importance of var in JavaScript

2008-09-04T00:37:00Z

I would consider myself proficient at JavaScript. Not a rockstar, but I can hold my own. But I didn’t learn it out of a book; I picked it up slowly over several years. So occasionally I come across something that I probably would have learned in the first couple chapters of a decent JavaScript book.

Today, I experienced the importance of declaring variables with var.

For the unenlightened like myself, JavaScript basically has two scopes for variables: global and local. Variables assigned outside of a function are global, and variables assigned inside of a function, using the var keyword, are local (not rocket surgery). However, if you leave the var keyword off, it assigns a global variable, regardless of where it’s declared.

Here’s a bit of code that demonstrates this:

function fail() {
  date = new Date();
  console.log('Before: ' + Number(date));
  setTimeout(function() {
    console.log('After: ' + Number(date));
  }, 1000);
}

fail();
fail();

The code defines a function that sets a variable, logs the variable, then defines a callback that logs the variable again on second later. If you call the function twice in a row, you can see the effect:

Before: 1220487263486
Before: 1220487263499
After: 1220487263499 <- This should be 1220487263486
After: 1220487263499

Declaring the date variable with var gives us this output:

Before: 1220487287985
Before: 1220487287994
After: 1220487287985 <- see, this is the same as the first line
After: 1220487287994

So remember kids, declare local variables with var!

Autotest mapping for Rails test conventions

2008-08-22T11:00:00Z

A while ago I posted a configuration for getting autotest to work with Test::Unit outside of Rails. Ryan Davis, author of autotest, commented on that post saying that it should “Just Work™” without any custom configuration. I was perplexed because I’ve never been able to get it to work on my gems and Rails plugins.

I finally took time to look into the issue, and realized it’s because I always use the Rails naming conventions for my test files. I name them foo_test.rb, instead of test_foo.rb, which is what Autotest looks for.

That’s easily solvable. Here’s an Autotest configuration, tested with ZenTest 3.10.0, that should make it work for either naming convention. You can throw this in your ~/.autotest file, or in a .autotest file inside your project.

Autotest.add_hook :initialize do |at|
  at.clear_mappings

  at.add_mapping %r%/^lib/(.*)\.rb$% do |_, m|
    possible = File.basename(m[1])
    files_matching %r%^test/.*(#{possible}_test|test_#{possible})\.rb$%
  end

  at.add_mapping(%r%^test/.*\.rb$%) {|filename, _| filename }
end

Happy autotesting.