Development
Cloud tasks for Ruby and Rails
Cloudtasker is an easy way to use Ruby framework allowing your app to leverage Google Cloud Tasks for background jobs. Here is how to get started.
In April last year Google released Cloud Run, a service helping companies move to serverless by providing a generic deployment methodology based on Docker.
For those not familiar with the concept, serverless computing is a runtime model where the scalability of applications is dynamically managed by the underlying infrastructure based on the actual load. Forget servers and think about dynamically spawning workers/threads based on the number of concurrent HTTP requests hitting your app at a given point in time. Beyond the scalability aspect this also means no idle servers and scale-down to zero instance, which may represent huge cost savings in a micro-service context.
Laurent Julliard (Google Engineering) wrote an excellent four-part tutorial explaining how to move a Ruby on Rails application to Cloud Run. The tutorial covers all fundamental aspects but one question remained unanswered: how to manage background jobs on Cloud Run using Ruby/Rails? If you ask yourself the same question, read on!
The Challenge
In a serverless context there is no active process waiting for jobs. Processes are spawned based on HTTP activity. This means that background servers like Sidekiq, Resque or Delayed::Job are not an option unless you're fine with maintaining a separate dedicated server for background jobs.
The Google Cloud response to running tasks in the cloud is Cloud Tasks. Google Cloud Tasks manages queues for you and sends webhooks to your application when a background job is due for execution.
Sounds like exactly what we need in a serverless context! But wait! Does it mean I have to re-implement Sidekiq using Cloud Tasks then? Don't worry you don't.
Introducing Cloudtasker
At Keypup we believe in Google Cloud and serverless. And because we make heavy use of background jobs for our data processing pipelines we decided to release Cloudtasker, a framework that makes it easy to plug your Ruby/Rails application to Cloud Tasks.
If you like Sidekiq for server-based job processing then you will love Cloudtasker for serverless jobs.
Show me already!
Alright. Here are the steps to get started with a Rails application.
Add the following to your Gemfile and run bundle install
In development it is annoying to have to connect to Cloud Tasks because you need to have a publicly accessible endpoint to receive webhooks from Cloud Tasks (e.g. using ngrok). Fortunately Cloudtasker ships with a local development server that mimics Google Cloud Tasks.
The only requirement for this local server to run is to install redis. E.g. with brew: brew install redis
Add the following initializer to your application:
Now define a Cloudtasker worker in your application:
Start your Rails application and the Cloudtasker local server:
Open a Rails console and enqueue some jobs:
Voila! You should now see your jobs getting processed by your Rails app in your logs:
Production setup
In production your application will use Google Cloud Tasks instead of the Cloudtasker local development server.
Edit your Cloudtasker initializer like this. Make sure to edit the GCP related settings as well as the public endpoint your app uses:
Create the default queue in Cloud Tasks using the gcloud sdk or the built-in rake task:
That's it! You can now deploy your application to Cloud Run or App Engine*. Your application will push jobs to Cloud Tasks when enqueuing workers and receive jobs for processing via webhooks from Cloud Tasks.
* Note: you may need to adjust the IAM permissions of the Cloud Run / App Engine service agent if these don't have admin permission on Cloud Tasks.
There is more to it
That was only a peek preview of Cloudtasker. The library offers many other options and features that may be useful to you such as:
- Configuring multiple queues
- Adding custom logging
- Handling job failures and customizing retries
- Managing cron jobs**
- Ensuring jobs are unique by argument**
- Defining batch jobs (track multiple jobs as one)**
All of them are explained in the Cloudtasker documentation. Happy background-jobbing!
** Note: these extensions require Redis. Unfortunately Cloud Run does not offer a connection to Memory Store yet. Only apps deployed to App Engine may use these features. Google Cloud plans to release serverless connectors between Cloud Run and Memory Store in the near future.
