December 24, 2010

Posted by John

Tagged clogging, harmony, and testing

Older: Hunt, An Experiment in Search

Newer: A Scam I Say!

Improving Your Methods

I am always looking for ways to improve my efficiency while coding. One of the things that has been bothering me lately is how I run tests. Back in the day, I used autotest. Of late, I have been using watchr. Finally, this week I worked out something that does exactly what I want.

Watchr

Watchr is great at running arbitrary code when files change, but it cannot read my mind. When working on libraries, such as Hunt, Joint, MongoMapper, etc., running tests/specs/features every time a file changes is fine. Heck, the whole suite MongoMapper runs in like 10 seconds.

You can take it a step further and make watchr even more responsive by making it run only related files. For example, whenever lib/mongo_mapper/plugins/accessible.rb is saved, most likely I want to run the test/functional/test_accessible.rb test (checkout MM’s specs.watchr file on Github).

The Problem

The issues I have had with watchr is more when working on big applications, such as Harmony or Words with Friends. Bigger applications have bigger test suites and the process of automatically determining which tests I want to run when a file changes is difficult to impossible and changes from moment to moment.

After about a month of the pain that is manually running specific test files, I finally decided to think out what I wanted. The conclusion I came to was that I do not want tests to automatically run when I save a file. Instead, I want it to be easier to run specific tests based on keywords.

The Solution

Since I was working on Harmony at the time and it uses test/unit, I decided to write a script that would grep through the tests to match arguments I pass in and run all of the matches. Below is the script I came up with:

#!/usr/bin/env ruby
require File.dirname(__FILE__) + '/../config/boot'

if ARGV.empty?
  puts <<-EOH
Usage:
  script/test [test match string]

Example:
  script/test site
  script/test site_test
  script/test site_test account_test

  Above would run all tests matching site
  (ie: test/unit/site_test.rb, test/functional/admin/sites_controller.rb, etc.)

EOH
  exit
end

tests = Dir.glob(File.dirname(__FILE__) + '/../test/**/*').map do |file|
  # skip non test files
  next unless file.include?('_test')

  # check if any of the inputs match the file
  if ARGV.any? { |match| file =~ /#{match}/ }
    File.expand_path(file).gsub(Rails.root.join('test'), 'test')
  end
end.compact.join(' ')

test_loader = File.expand_path('../test_loader', __FILE__)

$stdout.sync = true
command = "bundle exec ruby -Itest #{test_loader} #{tests}"

puts(command)

IO.popen(command) { |com| com.each_char { |c| print(c) } }

Note that it uses this test_loader, stolen directly from rake.

#!/usr/bin/env ruby

ARGV.each { |f| load f unless f =~ /^-/  }

I dropped both of these files in the script/ directory of Harmony. The cool thing is with not very much code, I can now do the following:

# run all of harmony's tests for liquid filters
# filters are all in a folder test/unit/filters
script/test filters

# same thing but drops
script/test drops

# run all the item related tests (from controllers, models, filters and drops)
script/test item

# run unit tests for feed and feed template
script/test unit/feed

# run unit test for feed and feed drop test
script/test unit/feed feed_drop

Conclusion

In general I am working on a very specific thing. All I want is to be able to test that very specific thing, on the fly, and very easily. script/test allows me to do that. Oh, and of course I aliased script/test to st, as I hate typing more than 2 characters. ;)

Not sure that this is valuable to anyone else as it is so specific to how I work, but it is the thought that counts. Also, It could easily be adapted to rspec or cucumber. I can say that it has drastically helped me.

I think the lesson to grep from this post is always be conscious of how you work and when you see a chance for improvement, spend a few minutes to make it happen.

6 Comments

  1. Bruce Hauman Bruce Hauman

    Dec 25, 2010

    How about tagging your test files at the top in a comment block and then autotest files that have those tags?

  2. I recently read Eric Ries’s chapter “Continuous Deployment” in the new book Web Operations. In it, he mentions having a tag-based test suite at IMVU to empower his team to do multiple deploys a day. I know this isn’t exactly the same thing (or even the same thing Bruce is suggesting), but this example is very timely, John. Thanks for posting it!

  3. A while ago, I have created a simple command-line tool to do exactly the same: https://github.com/mislav/polyamory

    I have big plans for it, so watch this space.

  4. @Mislav: Ha. Great minds think alike.

  5. Hi John, have you tried Guard instead of Watchr? It would be great to have this idea as an option for guard-test/rspec/cucumber. Thx!

  6. Erik Ostrom Erik Ostrom

    Jan 11, 2011

    I often have to prod autotest into running a test, but I’m only occasionally bothered by it running tests that weren’t necessary. So for me a good hybrid would be to keep using autotest, but write a script like yours that, instead of running the tests itself, touches the test files to make autotest rerun them.

Sorry, comments are closed for this article to ease the burden of pruning spam.

About

Authored by John Nunemaker (Noo-neh-maker), a programmer who has fallen deeply in love with Ruby. Learn More.

Projects

Flipper
Release your software more often with fewer problems.
Flip your features.