Skip to main content
Workflows run in containers orchestrated by Kubiya’s coordination platform - keeping execution isolated and reliable.

​
How it works

Container execution: Each workflow step runs in its own Docker container with specific tools and dependencies. Coordination: Kubiya’s platform manages workflow scheduling, step dependencies, and data flow between steps. State management: Step outputs are captured and passed to dependent steps automatically. Resource management: Containers are created, executed, and cleaned up automatically.

​
Key benefits

Isolation: Each step runs independently - no conflicts or shared state issues. Flexibility: Use any Docker image, language, or tool you need. Reliability: Same inputs always produce same outputs. Scalability: System scales automatically based on workload. Clean execution: Fresh environment for every step prevents drift and issues.

​
Simple example

from kubiya import workflow, step

@workflow
def process_data():
    # Step 1: Extract in Python container  
    extract = step("extract").docker(
        image="python:3.11",
        script="fetch_data.py"
    ).output("RAW_DATA")
    
    # Step 2: Transform in different container
    transform = step("transform").docker(
        image="apache/spark:latest", 
        command="spark-submit transform.py"
    ).depends("extract").output("CLEAN_DATA")
    
    # Step 3: Load using yet another container
    step("load").docker(
        image="postgres:15",
        script="load_to_db.py"
    ).depends("transform")

​
Execution flow

  1. Submit workflow: User submits workflow definition to Kubiya
  2. Schedule steps: Platform analyzes dependencies and schedules ready steps
  3. Create containers: Runner creates Docker container for each step
  4. Execute step: Container runs with inputs and captures outputs
  5. Clean up: Container is destroyed after execution
  6. Repeat: Process continues for next ready steps

​
Data flow

Step outputs: Each step can produce outputs that become available to dependent steps. Environment variables: Outputs are passed as environment variables to next steps. File artifacts: Steps can also share files through mounted volumes when needed. No shared state: Steps cannot interfere with each other’s execution environment.

​
Error handling

Step failures: Failed steps can be retried with configurable backoff strategies. Partial execution: Only affected steps need to re-run, not the entire workflow. Resource limits: Steps have CPU, memory, and time limits to prevent runaway processes. Clean recovery: Failed containers are removed and fresh ones created for retries.

​
Getting started

Define steps: Specify what containers and commands each step needs Set dependencies: Tell Kubiya which steps depend on others Handle outputs: Capture important data from steps for use in later steps Add error handling: Configure retries and timeouts for robust execution
Next: See practical workflow examples or learn the DSL syntax