Beginner

Fixing Rspec and Cucumber random failures in your Continuous Integration Service


When you have a large test suite chances are that some of the tests fail randomly and when this happens you are not going to want to re-run everything again. So here it is the configuration I always use for Rspec and Cucumber to re-run only those failing specs and features. We are using circleci in my current project.

This is how my circle.yml setup looks like:


machine: timezone: America/Mexico_City node: version: 5.11.0 environment: VARIABLE_SETUP_TIMEOUT: 50 dependencies: pre: - bundle exec rake prepare_assets test: override: - RAILS_ENV=test bundle exec rspec_runner:all_rspec: parallel: true files: - spec/controllers/**/*_spec.rb - spec/helpers/**/*_spec.rb - mkdir -p $CIRCLE_TEST_REPORTS/cucumber: parallel: true - bundle exec rake cucumber_runner:start: parallel: true files: - features/section_one/**/*.feature - features/section_two/**/*.feature - bundle exec rake cucumber_runner:rerun_failing_scenarios: parallel: true

For cucumber

Basically you need to indicate the rerun and a file to save failed features:


bundle exec cucumber rerun --out cucumber_failures.txt

This is how my rake task looks like:


namespace :cucumber_runner do FAILING_CUCUMBER_SCENARIOS_FILENAME = 'failing_scenarios.txt' desc 'Run cucumber tests in using the circle ci configuration' task :start do # delete previous failing cucumber test scenarios filename if File.exists?("#{FAILING_CUCUMBER_SCENARIOS_FILENAME}") File.delete("#{FAILING_CUCUMBER_SCENARIOS_FILENAME}") end # exit 0, we don't want to fail here exec("bundle exec cucumber --format junit --out $CIRCLE_TEST_REPORTS/cucumber/tests.cucumber -f rerun --out #{FAILING_CUCUMBER_SCENARIOS_FILENAME}; exit 0") end desc 'Re run cucumber scenarios that failed in the first intent' task :rerun_failing_scenarios do # we don't need to run cucumber again if all scenarios passed unless File.zero?("#{FAILING_CUCUMBER_SCENARIOS_FILENAME}") # run cucumber with failing scenarios only exec("bundle exec cucumber @#{FAILING_CUCUMBER_SCENARIOS_FILENAME} --format junit --out $CIRCLE_TEST_REPORTS/cucumber/tests.cucumber") end end task all_rspec: :environment do Rake::Task["spec:start"].execute Rake::Task["spec:retry"].execute end end

For Rspec

Basically, in order to re-run failed specs in the first run, you need to specify a file name in your Rspec setup, which will save the status of each spec in the first try and after that, you will be able to run only the ones that failed using the tag –only-failures:


bundle exec rspec # first try bundle exec rspec --only-failures # run only failures

Rspec setup:

RSpec.configure do |config|

config.example_status_persistence_file_path = "rspec_failures.txt"

end

Rake task for continuos integration software:
As you may know, whenever you run several commands in your CI if the first one fails, it won’t let you run the spec retry; to avoid this we need to catch the exception. You can see in the circle.yml configuration I’m running a custom rake task, and that looks like this:

content of Rakefile:

require File.expand_path('../config/application', __FILE__)

Rails.application.load_tasks

begin
require 'rspec/core/rake_task'
RSpec::Core::RakeTask.new("spec:start") do |t|
t.rspec_opts = "--format documentation"
t.verbose = false
t.fail_on_error = false # don't stop the whole suite
end

RSpec::Core::RakeTask.new("spec:retry") do |t|
t.rspec_opts = "--only-failures"
t.verbose = false
end
rescue LoadError
end

Afterward, you can easily run the whole rspec test suite and re-run only the ones that have failed on the first try:

bundle exec rake rspec_runner:all_rspec

Another alternative you can take is the following:

If you don’t want to dig deeper into your code and you need something that introduces the same concept -as I mentioned above- you can check this gem out: rspec-retry (This works also with capybara), I’ve tried this in some other project using codeship as CI server. I did not have problems with configurations or anything. I just followed this steps:

Add this line to your application’s Gemfile:

gem 'rspec-retry'

And then execute:

$ bundle install

require in spec_helper.rb

# spec/spec_helper.rb
require 'rspec/retry'

RSpec.configure do |config|
.
.
.

Usage

it 'should randomly succeed', retry: 3 do
expect(rand(2)).to eq(1)
end

it 'should succeed after a while', retry: 3, retry_wait: 10 do
expect(command('service myservice status')).to eq('started')
end

That’s it, You now have two options when you face this kind of problems, hope you find useful this post, keep in touch.

H

Beginner
Administrate review
Beginner
How to upgrade from rails 4.2.9 to rails 5.1 and don’t die trying
Community
Fear the nothing!