Cucumber Tests: Auto Retry Failed

If you're tired of having to re-kick builds in your CI server because of non deterministic failures by the cucumber tests, this post is for you. After I implemented this awesome feature our builds started to be more realistic, and developers actually care now if their branch is red.

I'm going to show you how I got to the point where failed cucumber features auto retry themselves. I started to dig into the cucumber documentation and I couldn't find how to implement an auto retry feature straightforward. After being a curious developer and reading some of the cucumber code I found that there was a formatter named 'rerun', this is so nice! I went back to the cucumber documentation and I found that it was almost what I was looking for and that I could use it straight in my cucumber profiles:

#config/cucumber.yml
selenium: --format pretty --format rerun --out tmp/rerun.txt features/selenium --require features/selenium/step_definitions --require features/selenium/support

It can receive an extra parameter --out, this is used to store failed tests along with the line number, awesome!!, isn't it?

Ok, now I had the ability to track failed tests and store them in a txt file, now what's next? I started to think how my rake task would look like, parsing the file and those ugly things, but I was sure that this problem had been already solved, it was as simple as run 'cucumber --help' in command line to find out, and guess what?

$ cucumber --help
Usage: cucumber [options] [ [FILE|DIR|URL][:LINE[:LINE]*] ]+

Examples:
cucumber examples/i18n/en/features
cucumber @rerun.txt (See --format rerun)
cucumber examples/i18n/it/features/somma.feature:6:98:113
cucumber -s -i http://rubyurl.com/eeCl

Whooohooo, it accepts the file generated by the rerun formatter!!! so now I tried this:

$ cucumber @tmp/rerun.txt --format pretty features/selenium \
--require features/selenium/step_definitions --require features/selenium/support

And, it worked!... it took failed tests and ran them! Cool, now I had to integrate this with our cucumber rake tasks.

I thought it'd be easy, so I started with a rerun rake task:

namespace :cucumber do
  Cucumber::Rake::Task.new(:rerun, 'Rerun failed cucumber tests') do |t|
    unless File.exist?(File.join(Rails.root, 'tmp/rerun.txt'))
    File.open(File.join(Rails.root, 'tmp/rerun.txt'), 'w+').close
    t.profile = 'rerun'
  end
end

Doh! I ran it and failed because I didn't specify the 'rerun' profile in cucumber.yml, easy to fix:

#config/cucumber.yml
selenium: --format pretty --format rerun --out tmp/rerun.txt features/selenium --require features/selenium/stepdefinitions --require features/selenium/support
rerun: @tmp/rerun.txt --format pretty features/selenium --require features/selenium/stepdefinitions --require features/selenium/support

Done, I got my 'rake cucumber:rerun' task and it worked just fine. My next step was to include this in 'rake cucumber:all' and, theoretically if the selenium profile failed, it'd execute rerun task and retry failed tests, but! it exited after the selenium profile failed and the task finished right away. Then I thought, of course! it works how it's supposed to work, it exits when it fails, so it was not that easy.

This was the tricky part, because I had to wrap this task up in a huge begin rescue block, I ended up implementing my own super sophisticated task runner:

def run_rake_task(name)
  begin
    Rake::Task[name].invoke
    rescue Exception $e
    return false
  end
  true
end

Cool, it wraps up my tasks and catches the error, I had everything I needed, so I created another rake task:

namespace :cucumber do
  desc 'Run selenium and rerun failed tests'
  task :selenium_with_retry do
    selenium_successful = run_rake_task("cucumber:selenium")
    rerun_successful = true
    unless selenium_successful
      rerun_successful = run_rake_task("cucumber:rerun")
    end
    unless selenium_successful || rerun_successful
      raise CucumberFailure.new 'Cucumber tests failed'
    end
  end
end

Done! I just replaced this new task in our main rake cruise task, and bingo! It now auto retries failed cucumber tests.

I know it has a lot of opportunity areas, but hopefully this helps to somebody else to get their builds green!

Thanks for reading!

Know more about us!