How to run long running tasks

You have a task that takes long time to accomplish. Perhaps it operates on large volumes of data, making tons of IO or just needs to wait long time for resources.

Of course doing the work in the controllers themself is not Ideal. Even if your request timeout is long enough (and it might not be), you don’t want to keep you clients waiting for responses and your sever occupied with stale requests.

Here is a list of patterns you can use to offload the work and free your controllers from the long running tasks:

Event Emitter

The event emitter is a NodeJS class that provides a simple messaging mechanism. It allows you to dispatch and listen to messages and by then freeing your controller from a task.

import { EventEmitter } from "events";

const eventEmitter = new EventEmitter();

const eventName = "start-doing-some-heavy-work";

app.post("some-route", (req, req) => {
  // ... validate the request and perform business logic

  emitter.emit(eventName);

  res.send(200);
});

eventEmitter.on(eventName, () => {
  // ... do the heavy work
});

** https://nodejs.org/api/events.html

Cron Job

Crontab is a linux command line utility for job scheduling. It lets you run scripts on a predefined schedule on a single machine.

Assuming you have a job.js file containing an executable script here is how to set a cronjob to run it:

Make sure cron is installed

sudo apt-get update
sudo apt-get install cron

Edit the user’s crontab using the crontab -e command
Add a new line to specify the cron schedule. The format is as follows:
Terminal window
```
* * * * * /usr/bin/node /path/to/your/job.js
```
Replace the placeholders:
- The five asterisks represent the schedule (minutes, hours, days, months, and days of the week).
- /usr/bin/node is the path to your Node.js binary (you can find it by running which node).
- /path/to/your/job.js should be the full path to your job.js script.
Save the crontab file and exit the editor.
The cron job is now configured and will run your job.js script at the specified schedule.

** For help with the scheduling asterix syntax checkout Crontab guro

In order to run tasks for specific clients it might be helpful to mark them in the database in someway and then, when running the job, lookup only the clients that need handling.

Scheduling libraries

There are many Node.js libraries that implement scheduling in one way or another. One example is Node Schedule which lets you define your scheduled tasks from within your Node.js program:

import schedule from "node-schedule";

const job = schedule.scheduleJob("42 * * * *", () => {
  // run your schduled job
});

Here is short list of scheduling libraries:

Using a queue

Another offloading pattern is using a message queue. It is a bit similar to using event emitter but requires using an additional program that manages the queues. Queueing is a whole topic for itself but if you want a relatively simple way to do it you can check out BullMQ. It is a Node.js library that implements queueing on top or Redis and is relatively simple to use.

I hope that now you have an initial way out of performing heavy and long running tasks in controllers and you understand how to leverage different mechanisms for doing it.

Let me know if one of these options worked for you :)