Beginner

Fixing Rspec and Cucumber random failures in your Continuous Integration Service


Sometimes when you have a large test suite chances are that some of them fail randomly and when this happen you would not want to re run everything again so here it is the configuration I used to have for Rspec and cucumber in order to re run only those failing specs and features, in my case in this project at MagmaLabs, we are using circleci.

This is how it looks my circle.yml setup:


machine: timezone: America/Mexico_City node: version: 5.11.0 environment: VARIABLE_SETUP_TIMEOUT: 50 dependencies: pre: - bundle exec rake prepare_assets test: override: - RAILS_ENV=test bundle exec rspec_runner:all_rspec: parallel: true files: - spec/controllers/**/*_spec.rb - spec/helpers/**/*_spec.rb - mkdir -p $CIRCLE_TEST_REPORTS/cucumber: parallel: true - bundle exec rake cucumber_runner:start: parallel: true files: - features/section_one/**/*.feature - features/section_two/**/*.feature - bundle exec rake cucumber_runner:rerun_failing_scenarios: parallel: true

For cucumber

Basically you need to indicate the rerun and a file to save failed features:


bundle exec cucumber rerun --out cucumber_failures.txt

This is how it looks my rake task:


namespace :cucumber_runner do FAILING_CUCUMBER_SCENARIOS_FILENAME = 'failing_scenarios.txt' desc 'Run cucumber tests in using the circle ci configuration' task :start do # delete previous failing cucumber test scenarios filename if File.exists?("#{FAILING_CUCUMBER_SCENARIOS_FILENAME}") File.delete("#{FAILING_CUCUMBER_SCENARIOS_FILENAME}") end # exit 0, we don't want to fail here exec("bundle exec cucumber --format junit --out $CIRCLE_TEST_REPORTS/cucumber/tests.cucumber -f rerun --out #{FAILING_CUCUMBER_SCENARIOS_FILENAME}; exit 0") end desc 'Re run cucumber scenarios that failed in the first intent' task :rerun_failing_scenarios do # we don't need to run cucumber again if all scenarios passed unless File.zero?("#{FAILING_CUCUMBER_SCENARIOS_FILENAME}") # run cucumber with failing scenarios only exec("bundle exec cucumber @#{FAILING_CUCUMBER_SCENARIOS_FILENAME} --format junit --out $CIRCLE_TEST_REPORTS/cucumber/tests.cucumber") end end task all_rspec: :environment do Rake::Task["spec:start"].execute Rake::Task["spec:retry"].execute end end

For Rspec

Basically in order to re run failed specs in the first run, you need to specify a file name in your Rspec setup, which will save the status of each spec in the first try and after that you are able to run only the ones that failed using the tag –only-failures:


bundle exec rspec # first try bundle exec rspec --only-failures # run only failures

Rspec setup:

RSpec.configure do |config|

config.example_status_persistence_file_path = "rspec_failures.txt"

end

Rake task for continuos integration software:
As you know when you run several commands in your CI if the first one fails that won’t let run the spec retry, in order to avoid that we need to catch the exception as you see in the circle.yml configuration I’m running a custom rake tasks and that looks like this:

content of Rakefile file:

require File.expand_path('../config/application', __FILE__)

Rails.application.load_tasks

begin
require 'rspec/core/rake_task'
RSpec::Core::RakeTask.new("spec:start") do |t|
t.rspec_opts = "--format documentation"
t.verbose = false
t.fail_on_error = false # don't stop the whole suite
end

RSpec::Core::RakeTask.new("spec:retry") do |t|
t.rspec_opts = "--only-failures"
t.verbose = false
end
rescue LoadError
end

Afterwards you can easily run the whole rspec test suite and re-run only the ones that failed in the first try:

bundle exec rake rspec_runner:all_rspec

Another alternative you can use

If you don’t want to dig deeper into your code and you need something that introduces the same concept as I mentioned above you can check this gem out: rspec-retry (This works also with capybara), I tried this in another project using codeship as CI server, I did not have problems with configurations or anything else, just followed the next steps:

Add this line to your application’s Gemfile:

gem 'rspec-retry'

And then execute:

$ bundle install

require in spec_helper.rb

# spec/spec_helper.rb
require 'rspec/retry'

RSpec.configure do |config|
.
.
.

Usage

it 'should randomly succeed', retry: 3 do
expect(rand(2)).to eq(1)
end

it 'should succeed after a while', retry: 3, retry_wait: 10 do
expect(command('service myservice status')).to eq('started')
end

That’s it, You now have two options when you face this kind of problems, hope you find useful this post, keep in touch.

H

Rails
Active Job
Best Practices
De Código, Café y Cervezas 07 – ¿Somos profesionales?
Community
De Código, Café y Cervezas 06 – ActiveModel::Serializer