Auto Retry Failed Cucumber Tests

If you’re tired of having to re-kick builds in your CI server because of non deterministic failures, this post is for you. After I implemented this awesome feature our builds started to be more realistic, and developers actually care now if their branch is red.

I’m going to show you how I got to the point where failed cucumber features auto retry themselves. I started to dig into the cucumber documentation and I couldn’t find how to implement an auto retry feature straightforward. After being a curious developer and reading some of the cucumber code I found that there was a formatter named ‘rerun’, this is so nice! I went back to the cucumber documentation and I found that it was almost what I was looking for and that I could use it straight in my cucumber profiles:

selenium: --format pretty --format rerun --out tmp/rerun.txt features/selenium --require features/selenium/step_definitions --require features/selenium/support

It can receive an extra parameter –out, this is used to store failed tests along with the line number, awesome!!, isn’t it?

Ok, now I had the ability to track failed tests and store them in a txt file, now what’s next? I started to think how my rake task would look like, parsing the file and those ugly things, but I was sure that this problem had been already solved, it was as simple as run ‘cucumber –help’ in command line to find out, and guess what?

$ cucumber --help
Usage: cucumber [options] [ [FILE|DIR|URL][:LINE[:LINE]*] ]+

cucumber examples/i18n/en/features
cucumber @rerun.txt (See --format rerun)
cucumber examples/i18n/it/features/somma.feature:6:98:113
cucumber -s -i

Whooohooo, it accepts the file generated by the rerun formatter!!! so now I tried this:

$ cucumber @tmp/rerun.txt --format pretty features/selenium \
--require features/selenium/step_definitions --require features/selenium/support

And, it worked!… it took failed tests and ran them! Cool, now I had to integrate this with our cucumber rake tasks.

I thought it’d be easy, so I started with a rerun rake task:

namespace :cucumber do, 'Rerun failed cucumber tests') do |t|
    unless File.exist?(File.join(Rails.root, 'tmp/rerun.txt')), 'tmp/rerun.txt'), 'w+').close
    t.profile = 'rerun'

Doh! I ran it and failed because I didn’t specify the ‘rerun’ profile in cucumber.yml, easy to fix:

selenium: --format pretty --format rerun --out tmp/rerun.txt features/selenium --require features/selenium/stepdefinitions --require features/selenium/support
rerun: @tmp/rerun.txt --format pretty features/selenium --require features/selenium/stepdefinitions --require features/selenium/support

Done, I got my ‘rake cucumber:rerun’ task and it worked just fine. My next step was to include this in ‘rake cucumber:all’ and, theoretically if the selenium profile failed, it’d execute rerun task and retry failed tests, but! it exited after the selenium profile failed and the task finished right away. Then I thought, of course! it works how it’s supposed to work, it exits when it fails, so it was not that easy.

This was the tricky part, because I had to wrap this task up in a huge begin rescue block, I ended up implementing my own super sophisticated task runner:

def run_rake_task(name)
    rescue Exception $e
    return false

Cool, it wraps up my tasks and catches the error, I had everything I needed, so I created another rake task:

namespace :cucumber do
  desc 'Run selenium and rerun failed tests'
  task :selenium_with_retry do
    selenium_successful = run_rake_task("cucumber:selenium")
    rerun_successful = true
    unless selenium_successful
      rerun_successful = run_rake_task("cucumber:rerun")
    unless selenium_successful || rerun_successful
      raise 'Cucumber tests failed'

Done! I just replaced this new task in our main rake cruise task, and bingo! It now auto retries failed cucumber tests.

I know it has a lot of opportunity areas, but hopefully this helps to somebody else to get their builds green!

Thanks for reading!

How to use internal redirects in AEM?
Best Practices
De Código, Café y Cervezas 07 – ¿Somos profesionales?
How to build Android apps and not crash in the attempt (Part I)