Development

Auto Retry Failed Cucumber Tests


If you’re tired of having to re-kick builds in your CI server because of non deterministic failures, this post is for you. After I implemented this awesome feature our builds started to be more realistic, and developers actually care now if their branch is red.

I’m going to show you how I got to the point where failed cucumber features auto retry themselves. I started to dig into the cucumber documentation and I couldn’t find how to implement an auto retry feature straightforward. After being a curious developer and reading some of the cucumber code I found that there was a formatter named ‘rerun’, this is so nice! I went back to the cucumber documentation and I found that it was almost what I was looking for and that I could use it straight in my cucumber profiles:

#config/cucumber.yml
selenium: --format pretty --format rerun --out tmp/rerun.txt features/selenium --require features/selenium/step_definitions --require features/selenium/support

It can receive an extra parameter –out, this is used to store failed tests along with the line number, awesome!!, isn’t it?

Ok, now I had the ability to track failed tests and store them in a txt file, now what’s next? I started to think how my rake task would look like, parsing the file and those ugly things, but I was sure that this problem had been already solved, it was as simple as run ‘cucumber –help’ in command line to find out, and guess what?

$ cucumber --help
Usage: cucumber [options] [ [FILE|DIR|URL][:LINE[:LINE]*] ]+

Examples:
cucumber examples/i18n/en/features
cucumber @rerun.txt (See --format rerun)
cucumber examples/i18n/it/features/somma.feature:6:98:113
cucumber -s -i http://rubyurl.com/eeCl

Whooohooo, it accepts the file generated by the rerun formatter!!! so now I tried this:

$ cucumber @tmp/rerun.txt --format pretty features/selenium \
--require features/selenium/step_definitions --require features/selenium/support

And, it worked!… it took failed tests and ran them! Cool, now I had to integrate this with our cucumber rake tasks.

I thought it’d be easy, so I started with a rerun rake task:

namespace :cucumber do
  Cucumber::Rake::Task.new(:rerun, 'Rerun failed cucumber tests') do |t|
    unless File.exist?(File.join(Rails.root, 'tmp/rerun.txt'))
    File.open(File.join(Rails.root, 'tmp/rerun.txt'), 'w+').close
    t.profile = 'rerun'
  end
end

Doh! I ran it and failed because I didn’t specify the ‘rerun’ profile in cucumber.yml, easy to fix:

#config/cucumber.yml
selenium: --format pretty --format rerun --out tmp/rerun.txt features/selenium --require features/selenium/stepdefinitions --require features/selenium/support
rerun: @tmp/rerun.txt --format pretty features/selenium --require features/selenium/stepdefinitions --require features/selenium/support

Done, I got my ‘rake cucumber:rerun’ task and it worked just fine. My next step was to include this in ‘rake cucumber:all’ and, theoretically if the selenium profile failed, it’d execute rerun task and retry failed tests, but! it exited after the selenium profile failed and the task finished right away. Then I thought, of course! it works how it’s supposed to work, it exits when it fails, so it was not that easy.

This was the tricky part, because I had to wrap this task up in a huge begin rescue block, I ended up implementing my own super sophisticated task runner:

def run_rake_task(name)
  begin
    Rake::Task[name].invoke
    rescue Exception $e
    return false
  end
  true
end

Cool, it wraps up my tasks and catches the error, I had everything I needed, so I created another rake task:

namespace :cucumber do
  desc 'Run selenium and rerun failed tests'
  task :selenium_with_retry do
    selenium_successful = run_rake_task("cucumber:selenium")
    rerun_successful = true
    unless selenium_successful
      rerun_successful = run_rake_task("cucumber:rerun")
    end
    unless selenium_successful || rerun_successful
      raise CucumberFailure.new 'Cucumber tests failed'
    end
  end
end

Done! I just replaced this new task in our main rake cruise task, and bingo! It now auto retries failed cucumber tests.

I know it has a lot of opportunity areas, but hopefully this helps to somebody else to get their builds green!

Thanks for reading!

AEM
How to use internal redirects in AEM?
Best Practices
De Código, Café y Cervezas 07 – ¿Somos profesionales?
Android
How to build Android apps and not crash in the attempt (Part I)