Queue Drivers and Tables

A driver decides where queued jobs wait until a worker runs them. Holo ships three: sync, database, and redis. You can define several connections in config/queue.ts and pick one per job.

Driver	Where jobs wait	Needs a worker	Extra package
sync	Nowhere, jobs run immediately	No	none
database	A jobs table in your database	Yes	@holo-js/queue-db
redis	Redis	Yes	@holo-js/queue-redis

npx holo install queue --driver <name> adds the right package when it creates config/queue.ts. It won't overwrite your existing config/queue.ts.

Installed packages

When config/queue.ts exists and Holo can read it, the config decides which queue packages are installed. The --driver flag doesn't change that. A redis connection installs @holo-js/queue-redis. A database connection, or a failed-job store that isn't false, installs @holo-js/queue-db. A queue package the config doesn't use is removed. For example, take a config that only uses sync and sets failed: false, which is what the installer writes. npx holo install queue --driver redis on that config adds nothing, and it removes @holo-js/queue-db and @holo-js/queue-redis if they are installed. Without failed: false, the config still needs @holo-js/queue-db, so the installer keeps it.

If you switch drivers, edit the config and run npx holo install queue again. The installer adds the packages the config now needs, and reruns with no changes do nothing.

If Holo can't read config/queue.ts, the --driver flag adds the package for that driver, and Holo adds any package the config still needs. It never removes a queue package that is already installed.

Sync

import { defineQueueConfig } from '@holo-js/queue'

export default defineQueueConfig({
  default: 'sync',
  failed: false,
  connections: {
    sync: {
      driver: 'sync',
      queue: 'default',
    },
  },
})

dispatch() runs the job right away and waits for it to finish. Delays and retries are ignored, and an error in the job is thrown back to the caller. Use it for local development, tests, and work that is fast enough to do inside the request.

Option	Default	Description
queue	'default'	The queue name recorded on jobs.

Database

The database driver stores jobs in a table on one of your database connections. It works with SQLite, PostgreSQL, and MySQL 8.0.17 or later, and needs no extra service.

import { defineQueueConfig } from '@holo-js/queue'
import { env } from '@holo-js/config'

export default defineQueueConfig({
  default: env('QUEUE_CONNECTION', 'database'),
  failed: {
    driver: 'database',
    connection: 'default',
    table: 'failed_jobs',
  },
  connections: {
    database: {
      driver: 'database',
      connection: 'default',
      table: 'jobs',
      queue: 'default',
      retryAfter: 90,
      sleep: 1,
    },
  },
})

Option	Default	Description
connection	'default'	Database connection name from config/database.ts.
table	'jobs'	Table that holds waiting jobs.
queue	'default'	Default queue name for this connection.
retryAfter	90	Seconds before a job a worker took, but never finished, can be taken by another worker. Job timeouts must be shorter than this. Must be at least 1.
sleep	1	Seconds a worker waits before checking an empty queue again. At most 2147483.
afterCommit	false	Queue every job on this connection only after the surrounding database transaction commits. See Dispatch after a database commit.

Create the jobs table

npx holo queue:table
npx holo queue:failed-table
npx holo migrate

queue:table writes a migration for the table of every database connection in your config, or for jobs if there are none. queue:failed-table writes one for the failed jobs table. They are regular migrations in your project, so commit them.

The full setup from scratch:

npx holo install queue --driver database
npx holo queue:table
npx holo queue:failed-table
npx holo migrate
npx holo queue:work

Run several workers

You can run as many workers as you need against the same jobs table. Two workers never take the same job at the same time. A job a worker took but didn't finish within retryAfter seconds can be taken again, so keep retryAfter longer than your slowest job. On MySQL and PostgreSQL, a worker skips jobs another worker is taking and moves on to the next one, so workers don't wait on each other.

On SQLite, workers take jobs one at a time, because SQLite allows one write at a time.

Redis

The redis driver keeps jobs in Redis, using BullMQ. Workers wait on Redis for new jobs instead of polling a table. Use it when you run Redis already or process many jobs.

import { defineQueueConfig } from '@holo-js/queue'
import { env } from '@holo-js/config'

export default defineQueueConfig({
  default: env('QUEUE_CONNECTION', 'redis'),
  failed: false,
  connections: {
    redis: {
      driver: 'redis',
      queue: 'default',
      retryAfter: 90,
      blockFor: 5,
    },
  },
})

The Redis server itself is set in config/redis.ts, which the installer creates. The queue uses its default Redis connection unless you name another one with connection:

import { env } from '@holo-js/config'
import { defineRedisConfig } from '@holo-js/kernel'

export default defineRedisConfig({
  default: env('REDIS_CONNECTION', 'default'),
  connections: {
    default: {
      url: env('REDIS_URL') || undefined,
      host: env('REDIS_HOST', '127.0.0.1'),
      port: env('REDIS_PORT', 6379),
      username: env('REDIS_USERNAME'),
      password: env('REDIS_PASSWORD'),
      db: env('REDIS_DB', 0),
    },
  },
})

When url is set, it's used instead of host and port. See Configuration for Redis Cluster and other connection styles.

Option	Default	Description
connection	the default in config/redis.ts	Redis connection name from config/redis.ts.
queue	'default'	Default queue name for this connection.
afterCommit	false	Queue every job on this connection only after the surrounding database transaction commits. See Dispatch after a database commit.
retryAfter	90	Seconds before a job held by a worker that stopped can be taken by another worker. A running worker keeps renewing its hold, so long jobs aren't handed out twice. Must be between 1 and 2147483 (about 24 days).
blockFor	5	Seconds a worker waits on Redis for a new job before checking again.
redis	none	Override Redis settings for this connection only: url, host, port, username, password, db, or clusters.

Redis Cluster only supports database 0, so setting db to anything else with clusters is an error.

Multiple connections

Define as many connections as you need and pick one per job or per dispatch:

import { defineQueueConfig } from '@holo-js/queue'

export default defineQueueConfig({
  default: 'database',
  failed: false,
  connections: {
    database: {
      driver: 'database',
      queue: 'default',
    },
    redis: {
      driver: 'redis',
      queue: 'default',
    },
  },
})

await SendDigest.dispatch({ userId: 42 }).onConnection('redis')

Each connection needs its own worker: npx holo queue:work --connection redis.

Plugin driver authors: set acceptsJobId: true only if dispatch() stores the job under job.id and returns that same id. afterCommit (on a dispatch or a connection) needs it, and is rejected for other drivers.

Queued image conversions

Media library conversions marked with .queued() are generated by a queue job instead of during the upload. See Image conversions.

The conversion job runs on the default connection and always uses the media queue. With the sync driver it runs right away. With database or redis, make sure a worker listens on media:

npx holo queue:work --queue default,media

WARNING

A worker started without --queue only listens on the connection's queue (usually default), so queued conversions would never run.

Next steps

• Running workers
• Failed jobs
• Environment variables