Kubernetes Setup for Runners

This guide provides in-depth instructions for deploying and configuring Kubiya runners on Kubernetes environments. Whether you're using a managed service like EKS, GKE, or AKS, or a self-hosted Kubernetes cluster, these steps will help you set up runners properly.

Kubernetes Requirements

Before deploying a Kubiya runner, ensure your Kubernetes environment meets these requirements:

Kubernetes version 1.16 or higher
A default StorageClass configured (for persistent storage)
RBAC enabled
Outbound internet access from the cluster
At least 1 vCPU and 2GB memory available

Preparation

Create a Namespace

Create a dedicated namespace for Kubiya components:

kubectl create namespace kubiya

Configure Service Account

Create a service account for the runner:

kubectl create serviceaccount kubiya-service-account -n kubiya

Set Up Registry Access

Create a secret for pulling images from Kubiya's registry:

kubectl create secret docker-registry kubiya-registry-secret \
  --namespace kubiya \
  --docker-server=registry.kubiya.ai \
  --docker-username=<provided-username> \
  --docker-password=<provided-password>

Deployment Methods

Using Manifest Files

Create the following YAML files:

Create RBAC Configuration

Save this as kubiya-rbac.yaml:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: kubiya-runner-role
  namespace: kubiya
rules:
- apiGroups: [""]
  resources: ["pods", "services", "configmaps", "secrets"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["apps"]
  resources: ["deployments", "statefulsets", "daemonsets"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: kubiya-runner-rolebinding
  namespace: kubiya
subjects:
- kind: ServiceAccount
  name: kubiya-service-account
  namespace: kubiya
roleRef:
  kind: Role
  name: kubiya-runner-role
  apiGroup: rbac.authorization.k8s.io

Create Runner Deployment

Save this as kubiya-runner.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kubiya-runner
  namespace: kubiya
  labels:
    app: kubiya-runner
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kubiya-runner
  template:
    metadata:
      labels:
        app: kubiya-runner
    spec:
      serviceAccountName: kubiya-service-account
      containers:
        - name: kubiya-runner
          image: kubiya/runner:latest
          resources:
            limits:
              cpu: "1"
              memory: "1Gi"
            requests:
              cpu: "0.5"
              memory: "512Mi"
          env:
            - name: RUNNER_ID
              value: "<your-runner-id>"
            - name: RUNNER_SECRET
              value: "<your-runner-secret>"
            - name: KUBIYA_API_URL
              value: "https://api.kubiya.ai"
            - name: LOG_LEVEL
              value: "info"
          livenessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 30
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 5
      imagePullSecrets:
        - name: kubiya-registry-secret

Create Persistent Storage (Optional)

Save this as kubiya-storage.yaml:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: kubiya-runner-cache
  namespace: kubiya
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi

Apply Configurations

Apply all the configurations:

kubectl apply -f kubiya-rbac.yaml
kubectl apply -f kubiya-storage.yaml # If using persistent storage
kubectl apply -f kubiya-runner.yaml

Advanced Configurations

High Availability Setup

For production environments, consider a high-availability configuration:

# For Kubernetes manifest
spec:
  replicas: 3
  
# For Helm chart
replicaCount: 3
 
# Configure pod disruption budget
podDisruptionBudget:
  enabled: true
  minAvailable: 1

Resource Quotas

For better resource management, set up namespace quotas:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: kubiya-quota
  namespace: kubiya
spec:
  hard:
    requests.cpu: "4"
    requests.memory: 4Gi
    limits.cpu: "8"
    limits.memory: 8Gi
    pods: "10"

Network Policies

Secure your runner with network policies:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: kubiya-runner-network-policy
  namespace: kubiya
spec:
  podSelector:
    matchLabels:
      app: kubiya-runner
  policyTypes:
  - Egress
  egress:
  - to:
    - ipBlock:
        cidr: 0.0.0.0/0
        except:
        - 10.0.0.0/8
        - 172.16.0.0/12
        - 192.168.0.0/16
    ports:
    - port: 443
      protocol: TCP

Integration with Kubernetes Distributions

Amazon EKS

For EKS-specific deployments:

IRSA Integration:

Update the service account with IAM role annotation:
apiVersion: v1 kind: ServiceAccount metadata: name: kubiya-service-account namespace: kubiya annotations: eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/KubiyaRunnerRole
Node Group Selection:

Use node selectors to target specific node groups:
nodeSelector: eks.amazonaws.com/nodegroup: kubiya-workloads

Monitoring and Logging

Prometheus Integration

Enable Prometheus metrics for your runner:

prometheus:
  enabled: true
  serviceMonitor:
    enabled: true
    additionalLabels:
      release: prometheus

Logging Configuration

Configure logging to suit your environment:

logging:
  level: info
  format: json
  output: stdout

Troubleshooting

Common Issues and Solutions

Runner not connecting:
- Check credentials (RUNNER_ID and RUNNER_SECRET)
- Verify network connectivity to api.kubiya.ai
- Check pod logs using kubectl logs -n kubiya deployment/kubiya-runner
Permission issues:
- Verify RBAC configurations
- Check service account permissions
- Inspect logs for permission denied errors
Resource constraints:
- Monitor resource usage with kubectl top pods -n kubiya
- Consider increasing resource limits
Image pull errors:
- Verify registry secret is correctly configured
- Check image name and tag are correct

Next Steps

Configure runner permissions for your specific use cases
Deploy the Enforcer service for advanced policy control
Set up your first agent to work with your runner

Kubernetes Setup

On this page