Worker Queues

​

When to use

• Scale execution horizontally across many machines.
• Isolate workloads by environment, team, region, or hardware (e.g., GPU vs. CPU).
• Reach private resources by attaching self-hosted workers inside your network.

​

What you’ll see in the Dashboard

• KPI chips: Active Workers • Queues • Capacity • Last Updated.
• Queue cards (one per queue):
  • Status (Active/Idle) and a capacity bar (e.g., 1 / 10 workers).
  • Connect Worker area with a ready-to-copy command.
  • Configure (edit name/description, control plane, max workers).
  • View workers attached to this queue.
• New Queue: creates another queue and gives you a fresh connection command.

​

How it works

1. A Control Plane provides context, tools, and LLM settings for execution.
2. Queues receive workflow steps that need to run.
3. Workers (your machines) connect to a queue and pull steps to execute.
4. Each step runs in an isolated container with streaming logs and audit.
5. Multiple workers can attach to the same queue for parallel execution. A single machine can also attach to multiple queues if it has capacity.

​

Create a new queue

1. Click New Queue.
2. Set a Queue Name and optional Display Name/Description. Tip: name by purpose, e.g., staging, gpu-tasks, prod-eu.
3. Choose a Control Plane (Kubiya Cloud is the default, fully managed option).
4. Set Max Workers (the max concurrent worker processes for this queue).
5. Click Create Queue. You’ll get a Connect Worker command.

​

Add workers to a queue

1. On the queue card, open Connect Worker and copy the command.
2. Run it on any machine you want to contribute capacity from. Example format: kubiya worker start --queue-id <QUEUE_ID>
3. Repeat on more machines to scale out.
   • Each started worker process consumes one worker slot.
   • The same machine may join multiple queues if resources allow.

​

Good defaults & tips

• Start with one Kubiya Cloud queue to get going; add self-hosted queues later for private resources.
• Keep max workers modest at first; raise as your team’s throughput grows.
• Use separate queues for boundaries (prod vs. staging, region, GPU, high-risk tasks).
• Add a clear description so teammates know what belongs on the queue.

​

Security & isolation

• Steps execute in containers with the environment’s configured context, policies, and Skills.
• For self-hosted workers, restrict the host’s access to only what the queue needs.
• Rotate credentials and remove idle workers by stopping the worker process on that host.

​

Troubleshooting

• Worker won’t connect: verify Python 3 + CLI are installed; re-copy the exact command; check outbound network access.
• Jobs stuck in queue: ensure at least one online worker and that capacity isn’t zero.
• “No capacity” warnings: increase Max Workers or attach more machines.
• Slow runs: add more workers or split work across additional queues.