DocumentationConfiguration

Configuration

Pidgey configuration

Configure the Pidgey client and worker to match your needs.

Unified Configuration

Pidgey uses a single pidgey.config.ts file for both your application and the worker process.

1. Create the Config

pidgey.config.ts
import { defineConfig } from '@pidgeyjs/core';
 
export default defineConfig({
  // Shared config
  adapter: 'sqlite',
  filename: './pidgey.db',
 
  // Worker-specific config
  worker: {
    jobsDir: 'jobs',
    concurrency: 10,
  },
});

2. Initialize the Client

Import this config in your client initialization:

lib/pidgey.ts
import { Pidgey } from '@pidgeyjs/next';
import config from '../pidgey.config';
 
// Initialize the client with the shared config
export const pidgey = Pidgey(config);

Adapter Configuration

Configure your storage backend in pidgey.config.ts:

SQLite

export default defineConfig({
  adapter: 'sqlite',
  filename: './pidgey.db', // or ':memory:'
});

Options:

  • adapter: 'sqlite'
  • filename: Database file path (default: './pidgey.db')

PostgreSQL

export default defineConfig({
  adapter: 'postgres',
  connection: process.env.DATABASE_URL,
});

Options:

  • adapter: 'postgres'
  • connection: PostgreSQL connection string

Neon

Neon is a serverless Postgres platform that works great with Pidgey. SSL is auto-detected from your connection string.

export default defineConfig({
  adapter: 'postgres',
  connection: process.env.DATABASE_URL, // Neon connection string
});

Advanced Pool Configuration:

export default defineConfig({
  adapter: 'postgres',
  connection: process.env.DATABASE_URL,
  poolConfig: {
    max: 10, // Adjust based on your Neon plan limits
    idleTimeoutMillis: 10000,
    connectionTimeoutMillis: 2000,
  },
});

Neon free tier includes 20 connections. The queue adapter defaults to 10 connections, and the worker adapter defaults to 20. Adjust pool sizes if running multiple workers.

Serverless Functions:

For Edge/serverless functions, consider using Neon’s serverless driver with HTTP-based connections:

import { Pool, neonConfig } from '@neondatabase/serverless';
import ws from 'ws';
 
neonConfig.webSocketConstructor = ws;
 
export default defineConfig({
  adapter: 'postgres',
  connection: process.env.DATABASE_URL,
});

Supabase

Supabase provides managed Postgres with connection pooling. SSL is auto-detected from your connection string.

export default defineConfig({
  adapter: 'postgres',
  connection: process.env.DATABASE_URL, // Supabase connection string
});

Using Supabase Pooler:

For production, use Supabase’s connection pooler (port 6543) for better connection management:

export default defineConfig({
  adapter: 'postgres',
  // Use session mode for full Postgres compatibility
  connection: process.env.DATABASE_URL.replace(':5432', ':6543') + '?pgbouncer=true',
});

Direct Connection for Workers:

Workers benefit from direct connections (port 5432):

export default defineConfig({
  adapter: 'postgres',
  connection: process.env.DATABASE_URL, // Direct connection (port 5432)
});

Use Supabase Pooler for your application client and direct connections for workers by setting different environment variables.

Redis

export default defineConfig({
  adapter: 'redis',
  options: {
    host: 'localhost',
    port: 6379,
    password: process.env.REDIS_PASSWORD,
  },
});

Options:

  • adapter: 'redis'
  • options: Redis connection options
    • host: Redis host
    • port: Redis port (default: 6379)
    • password: Redis password (optional)

Job Configuration

Configure individual jobs when defining them:

export const myJob = pidgey.defineJob({
  name: 'my-job',
  handler: async (data) => {
    // Handler logic
  },
  config: {
    retries: 5,
    timeout: 60000,
    queue: 'critical',
  },
});

Options:

OptionTypeDefaultDescription
retriesnumber3Max retry attempts on failure
timeoutnumber300000Job timeout in milliseconds (5 minutes)
queuestringJob nameQueue name for grouping jobs

Global Defaults

Set defaults for all jobs in your config:

pidgey.config.ts
export default defineConfig({
  adapter: 'sqlite',
  filename: './pidgey.db',
  defaultJobOptions: {
    maxAttempts: 3,
    timeout: 60000,
  },
  worker: {
    jobsDir: 'jobs',
  },
});
Precedence: global defaults → job config → runtime options

Retries

Control how many times a job retries on failure:

config: {
  retries: 5, // Try up to 5 times
}

Failed jobs automatically retry with exponential backoff.

Timeouts

Set maximum execution time:

config: {
  timeout: 30000, // 30 seconds
}

Jobs that exceed the timeout are marked as failed and retried (if retries remain).

Custom Queues

Group related jobs into queues:

export const criticalEmail = pidgey.defineJob({
  name: 'critical-email',
  handler: async (data) => {
    /* ... */
  },
  config: {
    queue: 'critical', // High-priority queue
  },
});
 
export const newsletterEmail = pidgey.defineJob({
  name: 'newsletter-email',
  handler: async (data) => {
    /* ... */
  },
  config: {
    queue: 'low-priority', // Separate queue
  },
});

Run dedicated workers for each queue:

# Worker 1: Only critical jobs
pidgey worker dev --queue critical --concurrency 50
 
# Worker 2: Only low-priority jobs
pidgey worker dev --queue low-priority --concurrency 5

Environment-Based Configuration

Use environment variables within your pidgey.config.ts to switch configurations:

pidgey.config.ts
import { defineConfig } from '@pidgeyjs/core';
 
const adapter = process.env.JOB_ADAPTER || 'sqlite';
 
export default defineConfig(
  adapter === 'redis'
    ? {
        adapter: 'redis',
        options: {
          host: process.env.REDIS_HOST!,
          port: Number(process.env.REDIS_PORT || 6379),
        },
      }
    : adapter === 'postgres'
      ? {
          adapter: 'postgres',
          connection: process.env.DATABASE_URL!,
        }
      : {
          adapter: 'sqlite',
          filename: process.env.SQLITE_FILE || './pidgey.db',
        }
);

Then switch via environment:

# Development
JOB_ADAPTER=sqlite npm run dev
 
# Production
JOB_ADAPTER=postgres DATABASE_URL=postgres://... npm start
 
# Scale
JOB_ADAPTER=redis REDIS_HOST=redis.internal npm start

Worker CLI Options

Override worker behavior via CLI flags:

# Concurrency
pidgey worker dev --concurrency 50
 
# Polling interval
pidgey worker dev --poll 1000
 
# Specific queues
pidgey worker dev --queue emails --queue reports
 
# Combined
pidgey worker start --concurrency 100 --poll 5000 --queue critical

Available flags:

  • --concurrency — Max concurrent jobs
  • --poll — Polling interval in milliseconds
  • --queue — Process specific queues (repeatable)

Best Practices

Singleton Client

Export one client instance and import it everywhere:

lib/pidgey.ts
export const pidgey = Pidgey({
  /* ... */
});
jobs/send-email.ts
import { pidgey } from '@/lib/pidgey';
 
export const sendEmail = pidgey.defineJob({
  /* ... */
});

Separate Concerns

Different queues for different job types:

// Fast, high-priority
config: { queue: 'realtime', timeout: 5000 }
 
// Slow, low-priority
config: { queue: 'background', timeout: 300000 }

Run dedicated workers with appropriate concurrency:

# High-priority: more workers, faster polling
pidgey worker start --queue realtime --concurrency 100 --poll 100
 
# Low-priority: fewer workers, slower polling
pidgey worker start --queue background --concurrency 10 --poll 5000

Progressive Timeouts

Match timeout to expected job duration:

// API call: short timeout
config: {
  timeout: 10000;
} // 10 seconds
 
// Image processing: medium timeout
config: {
  timeout: 60000;
} // 1 minute
 
// Report generation: long timeout
config: {
  timeout: 300000;
} // 5 minutes

Retry Strategy

Adjust retries based on job type:

// Idempotent API call: more retries
config: {
  retries: 5;
}
 
// Payment processing: fewer retries
config: {
  retries: 2;
}
 
// Email sending: moderate retries
config: {
  retries: 3;
}

Next Steps

AdaptersOverview