Background job for uploading an image from a remote URL using paperclip and sidekiq

Sometimes, we need to upload images to our servers to avoid dependencies on third parties also, for instance, an RSS feed or an external API.

This process might take a long time with Paperclip depending on the weight of the original image and the number of sizes (large, small, thumb, etc.) that Paperclip has to create from the original image and also if the storage of Paperclip is on a local store or an external store like Amazon S3.

The best way to do this is by running on a background worker to improve the performance experience. To enqueue this process in a background job, we are going to use a Sidekiq tool.

Below this is shown step by step.

Versions

• ruby 2.5
• rails 5.1.5
• sidekiq 5.1.1
• paperclip 5.2.1
• aws-sdk 3.0.1

Add gems to project

Add the following lines to Gemfile

      . . .
  gem 'aws-sdk', '~> 3.0.1' 
  gem 'paperclip', '~> 5.2.1'
  gem 'sidekiq', '~> 5.1.1'
      . . .

Install gems with bundle

$ bundle install

Configure sidekiq

Add file configuration on config/sidekiq.yml

---
:verbose: false
:concurrency: 2

:timeout: 8

:queues:
 - critical
 - default
 - <%= `hostname`.strip %>
 - low

production:
 :concurrency: 2
staging:
 :concurrency: 2

Activate sidekiq as queue adapter on config/application.rb

module Project
  class Application < Rails::Application
        . . .
    config.active_job.queue_adapter = :sidekiq
        . . .
  end
end

Since sidekiq 5 we need to enable the asynchronous method called on initializer config/initializers/sidekiq.rb.

Sidekiq::Extensions.enable_delay!

Start sidekiq server

$ bundle exec sidekiq

Configure paperclip

File storage (local storage)

Add the following configuration to any of these environment files: config/environment.rb, config/environments/development.rb or config/environments/production.rb.

Rails.application.configure do
      . . .
  Paperclip.options[:command_path] = 'usr/local/bin/convert'
      . . .
end

S3 storage

To configure S3 storage like amazon S3 service, you need to create a new account using this page https://portal.aws.amazon.com/billing/signup#/start and set up the following configuration:

Configure the environment file

Rails.application.configure do
      . . .
  config.paperclip_defaults = {
    storage: :s3,
    s3_credentials: {
      s3_region: ENV.fetch('AWS_REGION'),
      bucket: ENV.fetch('BUCKET_NAME'),
      access_key_id: ENV.fetch('FULL_ACCESS_KEY'),
      secret_access_key: ENV.fetch('SECRET_FULL_ACCESS_KEY'),
      s3_host_name: "s3-#{ENV.fetch('AWS_REGION')}.amazonaws.com"
    }
  }
      . . .
end

Define initializer for aws on config/initializers/aws.rb

Aws::VERSION =  Gem.loaded_specs['aws-sdk'].version

Migrations

Let’s suppose we have a users table, we need to add the image attachment field to use it with paperclip, and a field to store the remote url.

Migration for paperclip

$ rails generate paperclip user image

Migration for remote url

$ rails generate migration add_image_remote_url_to_users image_remote_url:string

After running the migration, execute the following command to apply the migrations:

$ rails db:migrate

Integrate with model

Add the following configuration to the user model (app/models/user.rb).

class User < ApplicationRecord
  has_attachment_file :image, styles: { thumb: '100x100' }, default_url: 'noimage.jpg'
  validates_attachment_content_type :image, content_type: /\Aimage\/.*\z/
      . . .
end

Create the Job

The next step is to create a job that will be queued in the “default queue” (app/jobs/image_upload_job.rb).

class ImageDownloaderJob
  include Sidekiq::Worker

  sidekiq_options queue: :default

  def perform(id)
    users = User.all
    users.each do |user|
      if user.image.url.match('noimage') && user.image_remote_url.present?
        io = open(URI.parse(beer.photo_remote_url))
        user.image = io
        user.save!
      end
    end
  end
end

Create the action controller

Create the action controller to trigger the background job (app/controllers/users_controller.rb).

class UsersController < ApplicationController
      . . .
  def download_images
    ::ImageDownloaderJob.perform_async
    redirect_to users_path, flash: { notice: 'Process running' }
  end
      . . .
end

With this tools you can offer a better user experience by uploading images to your server or bucket (i.e. Amazon S3). It’s better to be in control of your own images than to depend on a third party API. I hope this post is helpful for you.

Go to MagmaLabs, to know more about us!