Cucumber Tests: Auto Retry Failed
If you're tired of having to re-kick builds in your CI server because of non deterministic failures by the cucumber tests, this post is for you. After I implemented this awesome feature our builds started to be more realistic, and developers actually care now if their branch is red.
I'm going to show you how I got to the point where failed cucumber features auto retry themselves. I started to dig into the cucumber documentation and I couldn't find how to implement an auto retry feature straightforward. After being a curious developer and reading some of the cucumber code I found that there was a formatter named 'rerun', this is so nice! I went back to the cucumber documentation and I found that it was almost what I was looking for and that I could use it straight in my cucumber profiles:
#config/cucumber.yml selenium: --format pretty --format rerun --out tmp/rerun.txt features/selenium --require features/selenium/step_definitions --require features/selenium/support
It can receive an extra parameter --out, this is used to store failed tests along with the line number, awesome!!, isn't it?
Ok, now I had the ability to track failed tests and store them in a txt file, now what's next? I started to think how my rake task would look like, parsing the file and those ugly things, but I was sure that this problem had been already solved, it was as simple as run 'cucumber --help' in command line to find out, and guess what?
$ cucumber --help Usage: cucumber [options] [ [FILE|DIR|URL][:LINE[:LINE]*] ]+ Examples: cucumber examples/i18n/en/features cucumber @rerun.txt (See --format rerun) cucumber examples/i18n/it/features/somma.feature:6:98:113 cucumber -s -i http://rubyurl.com/eeCl
Whooohooo, it accepts the file generated by the rerun formatter!!! so now I tried this:
$ cucumber @tmp/rerun.txt --format pretty features/selenium \ --require features/selenium/step_definitions --require features/selenium/support
And, it worked!... it took failed tests and ran them! Cool, now I had to integrate this with our cucumber rake tasks.
I thought it'd be easy, so I started with a rerun rake task:
namespace :cucumber do Cucumber::Rake::Task.new(:rerun, 'Rerun failed cucumber tests') do |t| unless File.exist?(File.join(Rails.root, 'tmp/rerun.txt')) File.open(File.join(Rails.root, 'tmp/rerun.txt'), 'w+').close t.profile = 'rerun' end end
Doh! I ran it and failed because I didn't specify the 'rerun' profile in cucumber.yml, easy to fix:
#config/cucumber.yml selenium: --format pretty --format rerun --out tmp/rerun.txt features/selenium --require features/selenium/stepdefinitions --require features/selenium/support rerun: @tmp/rerun.txt --format pretty features/selenium --require features/selenium/stepdefinitions --require features/selenium/support
Done, I got my 'rake cucumber:rerun' task and it worked just fine. My next step was to include this in 'rake cucumber:all' and, theoretically if the selenium profile failed, it'd execute rerun task and retry failed tests, but! it exited after the selenium profile failed and the task finished right away. Then I thought, of course! it works how it's supposed to work, it exits when it fails, so it was not that easy.
This was the tricky part, because I had to wrap this task up in a huge begin rescue block, I ended up implementing my own super sophisticated task runner:
def run_rake_task(name) begin Rake::Task[name].invoke rescue Exception $e return false end true end
Cool, it wraps up my tasks and catches the error, I had everything I needed, so I created another rake task:
namespace :cucumber do desc 'Run selenium and rerun failed tests' task :selenium_with_retry do selenium_successful = run_rake_task("cucumber:selenium") rerun_successful = true unless selenium_successful rerun_successful = run_rake_task("cucumber:rerun") end unless selenium_successful || rerun_successful raise CucumberFailure.new 'Cucumber tests failed' end end end
Done! I just replaced this new task in our main rake cruise task, and bingo! It now auto retries failed cucumber tests.
I know it has a lot of opportunity areas, but hopefully this helps to somebody else to get their builds green!
Thanks for reading!