Skip to main content

​
Overview

Work queues provide the infrastructure for distributing and executing tasks across workers. Kubiya uses Temporal for reliable, scalable task orchestration with automatic retries and failure handling.

​
Task Queues (Legacy)

Task queues were the original queuing mechanism in Kubiya. They provide:
  • Basic task distribution
  • FIFO execution
  • Simple worker assignment
Note: Task queues are being phased out in favor of worker queues for better scalability and features.

​
Worker Queues

Worker queues are the modern queuing system built on Temporal. They provide:
  • Horizontal scaling: Add workers dynamically based on load
  • Priority queues: Execute high-priority tasks first
  • Partitioning: Distribute tasks across multiple queue partitions
  • Observability: Track queue depth, processing time, and worker health
  • Durability: Tasks survive crashes and restarts
Queue types:
  • Default queue: General-purpose task execution
  • High-priority queue: For urgent tasks
  • Low-priority queue: For background jobs
  • Dedicated queues: For specific workloads or teams

​
Workers

Workers are the execution engines that process tasks from queues. They:
  • Poll queues for available tasks
  • Execute agent and workflow operations
  • Report heartbeats for health monitoring
  • Handle task failures and retries
Worker configuration:
  • Concurrency limits
  • Timeout settings
  • Resource allocations
  • Queue subscriptions

​
Queue Architecture

Task/Request
    ↓
Worker Queue (Temporal)
    ↓
Workers (polling)
    ↓
Agent/Workflow Execution
    ↓
Results

​
Common Operations

​
Create a Worker Queue

POST /api/v1/environments/{environment_id}/worker-queues
{
  "name": "high-priority-queue",
  "description": "Queue for high-priority agent tasks",
  "runner_name": "default",
  "max_workers": 10
}
Note: Worker queues are scoped to environments

​
List Worker Queues

# List all queues in organization
GET /api/v1/worker-queues

# List queues for specific environment
GET /api/v1/environments/{environment_id}/worker-queues

​
Get Worker Registration Command

GET /api/v1/worker-queues/{queue_id}/worker-command
Returns the command to start a worker for this queue.

​
Get Install Script

GET /api/v1/worker-queues/{queue_id}/install-script?platform=docker
Query parameters:
  • platform: docker, kubernetes, or local

​
Register a Worker

POST /api/v1/workers/register
{
  "worker_id": "worker-001",
  "queue_name": "org-id.default",
  "capabilities": {
    "runtime": "claude_code",
    "tools": ["kubernetes", "terraform"]
  }
}

​
Send Worker Heartbeat

POST /api/v1/workers/{worker_id}/heartbeat
{
  "status": "active",
  "queue_name": "org-id.default"
}
Or use the simplified endpoint: 
POST /api/v1/workers/heartbeat
{
  "worker_id": "worker-001",
  "status": "active"
}

​
Queue Monitoring

Monitor queue health through:
  • Queue depth (pending tasks)
  • Processing rate (tasks per minute)
  • Worker count (active workers)
  • Task latency (time to execution)
  • Error rates

​
Best Practices

  1. Queue Design: Create separate queues for different priorities or workload types
  2. Worker Scaling: Scale workers based on queue depth and latency
  3. Heartbeats: Send regular heartbeats to detect unhealthy workers
  4. Timeouts: Set appropriate task timeouts to prevent stuck tasks
  5. Monitoring: Track queue metrics to identify bottlenecks
  6. Resource Limits: Configure concurrency limits to prevent resource exhaustion

​
Migrating from Task Queues

If you’re using legacy task queues, migrate to worker queues for:
  • Better scalability and performance
  • Enhanced observability
  • More flexible worker management
  • Improved reliability with Temporal

​
Next Steps

Explore the API endpoints for worker queues, task queues, and worker management to set up your task orchestration infrastructure.