When you have a large test suite chances are that some of the tests fail randomly and when this happens you are not going to want to re-run everything again. So here it is the configuration I always use for Rspec and Cucumber to re-run only those failing specs and features. We are using circleci in my current project.
This is how my circle.yml setup looks like:
machine:
timezone:
America/Mexico_City
node:
version: 5.11.0
environment:
VARIABLE_SETUP_TIMEOUT: 50
dependencies:
pre:
- bundle exec rake prepare_assets
test:
override:
- RAILS_ENV=test bundle exec rspec_runner:all_rspec:
parallel: true
files:
- spec/controllers/**/*_spec.rb
- spec/helpers/**/*_spec.rb
- mkdir -p $CIRCLE_TEST_REPORTS/cucumber:
parallel: true
- bundle exec rake cucumber_runner:start:
parallel: true
files:
- features/section_one/**/*.feature
- features/section_two/**/*.feature
- bundle exec rake cucumber_runner:rerun_failing_scenarios:
parallel: true
For cucumber
Basically you need to indicate the rerun and a file to save failed features:
bundle exec cucumber rerun --out cucumber_failures.txt
This is how my rake task looks like:
namespace :cucumber_runner do
FAILING_CUCUMBER_SCENARIOS_FILENAME = 'failing_scenarios.txt'
desc 'Run cucumber tests in using the circle ci configuration'
task :start do
# delete previous failing cucumber test scenarios filename
if File.exists?("#{FAILING_CUCUMBER_SCENARIOS_FILENAME}")
File.delete("#{FAILING_CUCUMBER_SCENARIOS_FILENAME}")
end
# exit 0, we don't want to fail here
exec("bundle exec cucumber --format junit --out $CIRCLE_TEST_REPORTS/cucumber/tests.cucumber -f rerun --out #{FAILING_CUCUMBER_SCENARIOS_FILENAME}; exit 0")
end
desc 'Re run cucumber scenarios that failed in the first intent'
task :rerun_failing_scenarios do
# we don't need to run cucumber again if all scenarios passed
unless File.zero?("#{FAILING_CUCUMBER_SCENARIOS_FILENAME}")
# run cucumber with failing scenarios only
exec("bundle exec cucumber @#{FAILING_CUCUMBER_SCENARIOS_FILENAME} --format junit --out $CIRCLE_TEST_REPORTS/cucumber/tests.cucumber")
end
end
task all_rspec: :environment do
Rake::Task["spec:start"].execute
Rake::Task["spec:retry"].execute
end
end
For Rspec
Basically, in order to re-run failed specs in the first run, you need to specify a file name in your Rspec setup, which will save the status of each spec in the first try and after that, you will be able to run only the ones that failed using the tag --only-failures:
bundle exec rspec # first try
bundle exec rspec --only-failures # run only failures
Rspec setup:
RSpec.configure do |config|
config.example_status_persistence_file_path = "rspec_failures.txt"
end
Rake task for continuos integration software:
As you may know, whenever you run several commands in your CI if the first one fails, it won’t let you run the spec retry; to avoid this we need to catch the exception. You can see in the circle.yml configuration I’m running a custom rake task, and that looks like this:
content of Rakefile:
require File.expand_path('../config/application', __FILE__)
Rails.application.load_tasks
begin
require 'rspec/core/rake_task'
RSpec::Core::RakeTask.new("spec:start") do |t|
t.rspec_opts = "--format documentation"
t.verbose = false
t.fail_on_error = false # don't stop the whole suite
end
RSpec::Core::RakeTask.new("spec:retry") do |t|
t.rspec_opts = "--only-failures"
t.verbose = false
end
rescue LoadError
end
Afterward, you can easily run the whole rspec test suite and re-run only the ones that have failed on the first try:
bundle exec rake rspec_runner:all_rspec
Another alternative you can take is the following:
If you don’t want to dig deeper into your code and you need something that introduces the same concept -as I mentioned above- you can check this gem out: rspec-retry (This works also with capybara), I've tried this in some other project using codeship as CI server. I did not have problems with configurations or anything. I just followed this steps:
Add this line to your application's Gemfile:
gem 'rspec-retry'
And then execute:
$ bundle install
require in spec_helper.rb
# spec/spec_helper.rb
require 'rspec/retry'
RSpec.configure do |config|
.
.
.
Usage
it 'should randomly succeed', retry: 3 do
expect(rand(2)).to eq(1)
end
it 'should succeed after a while', retry: 3, retry_wait: 10 do
expect(command('service myservice status')).to eq('started')
end
That's it, You now have two options when you face this kind of problems, hope you find useful this post, keep in touch.
H
