Crails Framework

1. Introduction

Sidetasks are parts of your application that can run in a separate process, independently from the web application. They can be scheduled to run at a later time, and will persist even as the server restarts.

In this tutorial, we'll use the sidekix to see how we can add sidetasks in our application.

1.1 Guide assumptions

This guide will be using the sync plugin to follow the progress of sidetasks. You may want to take the Task Progression tutorial first.

1.2 Preparation

Create an application with the following command:

$ crails new -n scheduletask -p html
$ cd scheduletask

Then, add the sidekix and sync plugins:

$ crails plugins sidekix install
$ crails plugins sync install

Add a layout:

$ crails scaffold layout --toolkit bootstrap

Lastly, we'll use the task/show view from the Task Progression tutorial:

app/views/task/show.html

std::string @task_uri;
std::string @task_id;
// END LINKING
<h1>Task Watcher</h1>
<h3>Watching task <%= task_id %></h3>

<div class="progress">
  <div id="progress-bar" class="progress-bar" role="progressbar">
    0/0
  </div>
</div>

<script>
  const socket = new WebSocket("ws://0.0.0.0:3001/<%= task_uri %>");
  socket.onmessage = event => {
    const metadata = JSON.parse(event.data);
    const percents = metadata.progress * 100;
    const progressBar = document.querySelector("#progress-bar");

    progressBar.textContent = `${metadata.item_progress}/${metadata.item_count}`;
    progressBar.style.width = `${Math.ceil(percents)}%`;
  };
</script>

2. Sidetasks

2.1 Creating a sidetask

Let's first create a sidetask. The sidekix provides a command to set up new tasks:

$ crails plugins sidekix add-task --name my-sidetask

This will generate a source file for our task, which will export a function called my_sidetask.

We'll open that file and implement our side task:

tasks/sidekix/sidetasks/my_sidetask.cpp

#include <crails/params.hpp>
#include <crails/sync/task.hpp>
#include <crails/logger.hpp>
#include <unistd.h> // provides usleep

using namespace std;
using namespace Crails;

void my_sidetask(Crails::Params& params)
{
  string task_scope = "sidetask";
  string task_uid = params["sidekix"]["task_uid"];
  unsigned int step_count = 100;
  Sync::Task progress(task_scope, task_uid, step_count);

  logger << "Starting sidetask" << Logger::endl;
  while (step_count--)
  {
    usleep(25000);
    progress.increment();
  }
  logger << "Done with sidetask" << Logger::endl;
}

2.2 Invoking a sidetask

We shall now instructs sidekix to run our task. Create the following controller:

app/controllers/sidetask.hpp

#pragma once
#include "application.hpp"
#include <crails/sidekix.hpp>

class SidetaskController : public ApplicationController
{
public:
  SidetaskController(Crails::Context& context) :
    ApplicationController(context)
  {
  }

  void index()
  {
    std::string task_id;

    task_id = Sidekix::async_task("my-sidetask");
    vars["layout"] = std::string("layouts/application");
    vars["task_id"] = task_id;
    vars["task_uri"] = "sidetask/" + task_id;
    render("task/show");
  }
};

Let's create the matching routes for the index action and the sidetask/:task_uid channel we just defined in our controller:

app/routes.cpp

#include <crails/router.hpp>
#include <crails/sync/channel_actions.hpp>
#include "controllers/sidetask.hpp"

void Crails::Router::initialize()
{
  match_action("GET", "/sidetask", SidetaskController, index);
  match_sync_task("sidetask/:task_uid");
}

Build your server and go to http://localhost:3001/sidetask: you should witness the task progression.

2.3 Sending parameters to a sidetask

Sidetasks can also use parameters. And we will use that feature by setting the number of steps in our tasks in the controller. Open the controller file and update the index method as such:

app/controllers/sidetask.hpp

...
  void index()
  {
    std::string task_id;
    DataTree task_parameters;

    task_parameters["step_count"] = 42;
    task_id = Sidekix::async_task("my-sidetask", task_parameters);
    ...
  }
...

We've sent the step_count parameter to the asynchronous task: now let's use that parameter in the sidetask. Open the sidetask source file:

tasks/sidekix/sidetasks/my_sidetask.cpp

...
void my_sidetask(Crails::Params& params)
{
  string task_scope = "sidetasks";
  string task_uid = params["sidekix"]["task_uid"];
  // We can access our step_count parameter as such:
  unsigned int step_count = params["step_count"].as<int>();
  Crails::Sync::Task progress(task_scope, task_uid, step_count);

  while (step_count--)
  {
    usleep(250);
    progress.increment();
  }
}

3. Task scheduling

3.1 Schedule in

A big interest of asynchronous task is the ability to schedule those to run at a specific time. To do so, we replace async_task with schedule_task_in:

app/controllers/sidetask.hpp

#pragma once
#include <chrono> // add chrono for time operations
#include <crails/sidekix.hpp>
#include "application.hpp"

class SidetaskController : public ApplicationController
{
public:
  SidetaskController(Crails::Context& context) :
    ApplicationController(context)
  {
  }

  void index()
  {
    using namespace std::literals::chrono_literals;
    DataTree task_parameters;
    std::string task_id;

    task_parameters["step_count"] = 42;
    task_id = Sidekix::schedule_task_in(
      1m + 30s,
      "my-sidetask",
      task_parameters
    );
    vars["layout"] = std::string("layouts/application");
    vars["task_id"] = task_id;
    vars["task_uri"] = "sidetask/" + task_id;
    render("task/show");
  }
};

With this change, the task will run 10 seconds after the query is performed.

3.2 Schedule at

You may also want to schedule a task at a specific time. This is done by using the schedule_task function:

app/controllers/sidetask.hpp

...
  void index()
  {
    using namespace std::literals::chrono_literals;
    DataTree task_parameters;
    std::string task_id;

    task_parameters["step_count"] = 42;
    task_id = Sidekix::schedule_task(
      std::chrono::system_clock::now() + 10s,
      "my-sidetask",
      task_parameters
    );
    ...
  }
...

4. Settings

Using the config/sidekix.cpp file, you can configure the backends used by Sidekix to store pending tasks.

By default, the initialization code looks like this:

config/sidekix.cpp

...
static const string task_store_directory = ".pending-tasks";

SideTaskDatabase* SideTaskDatabase::initialize()
{
  if (Crails::environment == Crails::Test)
    return new Tests::Database();
  return new FileDatabase(task_store_directory);
}

It is a simple setup that configure a dummy database when running from the Test environment, or a file-based database otherwise.

4.1 File database

The default backend for Sidekix is a file-based backend: tasks are stored as files, and fetched from the filesystem by the task runner.

You can customize the path of the folder in which the tasks will be recorded as such:

...

SideTaskDatabase* SideTaskDatabase::initialize()
{
  return new Crails::Sidekix::FileDatabase("/usr/local/share/my-pending-tasks");
}

4.2 Redis database

If you've added a Redis database to your application with the following command:

$ crails plugins redis install

Then you may use the Redis backend for Sidekix tasks. The Redis backend is far more appropriate: when running multiple instances of your application on several machines, your sidetasks will be distributed accross all the instances, rather than only visible on the one instance that had scheduled them.

If you have a Redis database configured in your config/databases.cpp file, you can set up a Redis-backed sidetask database as such:

...
SideTaskDatabase* SideTaskDatabase::initialize()
{
  if (Crails::environment == Crails::Test)
    return new Tests::Database();
  return new RedisDatabase();
}

4.2.1 Configure the database used

RedisDatabase() will use the default redis database configured in your config/databases.cpp file. You can also setup a different configured database as such:

  return new RedisDatabase("alternative-redis-db");

4.2.2 Configure the storage key

By default, the sidetasks will be stored in a sorted list with the key sidekix-task, but you can also configure this:

  return new RedisDatabase("redis", "my-custom-key");

5. Testing

When writing your application tests, you might want to check whether your tasks have been scheduled properly. This is why a dummy backend is used in the test environment, which you can interact with by getting a pointer to the TestDatabase:

  auto* database = reinterpret_cast<Crails::Sidekix::TestDatabase*>(
    Crails::Sidekix::SideTaskDatabase::instance()
  );

With this pointer, you can perform several operations to check whether your tasks have been properly added:

  // Check that my-sidetask has been scheduled:
  EXPECT(database->has_task("my-sidetask"), ==, true);

  // Check that my-sidetask has been scheduled a specific amount of times
  EXPECT(database->count("my-sidetask"), ==, 12);

  // Check that my-sidetask has been scheduled at a specific time
  EXPECT(database->scheduled_at("my-sidetask", timestamp), ==, true))