Rails: Callback Hell and Pipeline Heaven   Let’s talk about the problem.

Node.js
Voiced by Amazon Polly

Clearly, many developers who’ve tried to code using Node.js — or even those who’ve just researched some information about the technology — know the meaning of “callback hell,” as well as the most common ways to get out of it.
For the purposes of illustrating callback hell, we will use an example containing pseudocode that looks similar to JavaScript:

function callApi(err, data) {
  api_request.send(data, function (err, result) {
    database.write(err, result, function (err, write_status) {
      response.render(write_status);
    })
  });
}

You will undoubtedly agree that something good wouldn’t be referred to as a “callback hell” or a “pyramid of doom.” Indeed, it’s pretty hard to find good things in this code. But it’s really easy to highlight all the problems:

  1. The readability of the code is simply terrible.
  2. It’s pretty hard to cover this code with tests.
  3. The debugging process can be used to scare not only kids but also all the world’s misbehaving people.

In this case, developers can use “Promises” to transform this knot of code vipers into an attractive and almost linear structure in which each new operation comes right after the previous one. But to do this, we need to return to Rails.
Ruby uses a different method of coding, so there’s no chance we can get back into callback hell, right? Sadly, callback hell can be found even in Rails, though it has a slightly different form and is included in the ActiveModel. I’m referring to after_create, before_save, after_destroy, and other similar elements. These callbacks are incredibly powerful tools that can transform into real problems when used in the wrong way.

What is our main enemy?

Usually, everything starts out pretty similarly: i.e., the coder has to make something with the operations on models. If a specific action must be performed every time we create a new entity, it’s pretty logical to use after_create for it. But let’s travel through time to move one year ahead in the life of the project. As with any development project, the code has become bigger, and several new developers have begun working on it. Since the business has also been growing, product owners sent new user stories, with the majority of them based on operations on the following models:

  • When I, as a user, invite another user…
  • When I, as an administrator, edit a group…
  • When I, as a user, leave a comment…
  • Dozens and dozens of other user stories

Many of these situations can be reflected in callbacks. (Frankly, we’re lucky that not all of them can be.) While it’s pretty difficult to track and test such changes, it’s really easy to make them. You simply need to find a specific callback, write a couple of strings, and you are glorious. The simple fact is that developers need to use this method to finish everything before the deadline, so we can’t blame them for it. But I want to pause here so that we can think about our direction.
These callbacks contain a lot of business logic, which (in the worst case) may include various notifications and even operations with other entities. In other words, SRP (single responsibility principle) — with which ActiveRecord has a lot of problems — is simply ignored almost entirely. The testing process becomes more and more complex because the code for testing can’t be divided from operations on models. The debugging process becomes an attempt to find the balance between “How can I call it?” and “Where did it come from?” As a result, we end up with a giant technical debt right in the heart of the project.

So what’s the solution?

It would be really ungentlemanly if I decided to write about these problems and disadvantages without offering an alternative. So I won’t do that. To start approaching a solution, let’s look at the business logic from a different point of view, forgetting about the callbacks in Rails. In doing that, we can easily note that, for example, the process of group editing is not an atomic action, but rather a process that contains several steps.

We call the process itself Pipeline, while each step is called PipelineNode. Thus, we need to create two relatively simple classes:

class Pipelines::Pipeline
  def initialize(nodes)
    @nodes = []
    nodes.each do |sym|
      node = Pipelines::Nodes.const_get(sym).new
      @nodes.last.next_node = node unless @nodes.empty?
      @nodes << node
    end
  end
  def process(args)
    @nodes[0].process(args) if @nodes.present?
  end
end

 

class Pipelines::PipelineNode
  attr_accessor :next_node
  def process(args)
  end
  def process_next(args)
    if next_node.present?
      next_node.process args
    else
      args[:model]
    end
  end
end

PipelineNode is a basic class for step description. Process method accepts Hash or an object with a set of data, while it has to return process_next args. All the data is stored in args, including (for example) a manipulated model or a user on behalf of whom we sent a request. Pipeline combines the steps together, indicating the next PipelineNode for the previous one. As we can see from the initialize method, PipelineNode must be described in the Pipelines: :Nodes module; however, it can be changed if required. First of all, we need to describe the required Pipelines for each model. As an example, I will take an average one from my project.

UPDATE_PIPELINE = [
  :CollectSubscribersNode,
  :AssignNode,
  :GroupBeforeSaveNode,
  :SaveNode,
  :GroupEmailNode,
  :GroupPushNode,
  :GroupNewsfeedNode
]

Part of PipelineNodes (e.g., GroupBeforeSaveNode) is created on the basis of slightly changed callbacks. Later, we need to divide them into smaller parts and find proper names to describe their content. For example, BeforeSave isn’t a great name because it doesn’t say what happens and only informs us about when.
From there, the rest is really easy: We simply need to combine everything and launch it.

args = {
  model: @group,
  params: groups_params,
  user: @current_user
}
Pipelines::Pipeline.new(Group::UPDATE_PIPELINE).process(args)

Moreso than other areas, the project core is the part that’s vulnerable to the technical debt. Bad technical solutions or mistakes made in the attempt to finish everything in time can make the developer’s life a real hell. Thus, the priority of such a technical debt is incredibly high, even though it’s really difficult to pay back.
So, with this solution, what do we have? Each and every step can be described separately. Each process method can be enhanced with specific checks so that we can be sure that all required data can be found in args. Furthermore, PipelineNode can be easily tested.
Conveyor has proven itself to be a great tool for mass production. This solution is built using the same principles, and unlike ActiveModel callbacks, it can be extended almost without limits.
Want to learn more about how Distillery’s developers find solutions that help us streamline the product development process? Let us know!

previous post next post
BACK TO TOP >