Source Management with Kubiya CLI

This guide provides comprehensive instructions for managing automation sources with the Kubiya CLI.

Understanding Sources

Sources are collections of tools, scripts, and configurations that define what your Kubiya teammates can do. Sources can be:

• Git repositories containing Kubiya tools

• Local directories with tool definitions

• External integrations and API connections

Listing Sources

To see all available sources in your organization:

kubiya source list

For more detailed output:

kubiya source list --full

For JSON format:

kubiya source list --output json

For more comprehensive details including metadata:

kubiya source list --full --fetch-metadata

Scanning for Sources

You can scan local directories to find potential Kubiya sources:

kubiya source scan ~/projects

With recursive scanning:

kubiya source scan ~/projects --recursive

Limit depth of recursion:

kubiya source scan ~/projects --recursive --max-depth 3

Adding Sources

Adding a Git Repository Source

kubiya source add --name "aws-tools" \
                 --url "https://github.com/organization/aws-tools" \
                 --branch "main"

Adding a Private Git Repository

kubiya source add --name "private-tools" \
                 --url "https://github.com/organization/private-repo" \
                 --auth-token "YOUR_ACCESS_TOKEN"

Adding a Local Directory Source

kubiya source add --name "local-tools" \
                 --path "/path/to/local/tools"

Adding a Source with Inline Tools

You can add a source with predefined tools directly:

# From a JSON file containing tool definitions
kubiya source add --name "inline-tools" --tools-file ./tools.json

# From stdin
cat tools.json | kubiya source add --name "inline-tools" --tools-stdin

Advanced Source Configuration

When adding a source, you can configure additional options:

kubiya source add --name "custom-source" \
                 --url "https://github.com/org/repo" \
                 --branch "develop" \
                 --auth-token "token" \
                 --disable-auto-sync \
                 --remote "upstream" \
                 --runner "my-runner"

Viewing Source Details

To get detailed information about a specific source:

kubiya source describe SOURCE_ID

This shows tools, integrations, and other metadata about the source.

With full details about each tool:

kubiya source describe SOURCE_ID --verbose

Syncing Sources

Syncing a Specific Source

kubiya source sync SOURCE_ID

Syncing All Sources

kubiya source sync --all

Syncing in Different Modes

Interactive mode (default):

kubiya source sync SOURCE_ID --mode interactive

Non-interactive mode:

kubiya source sync SOURCE_ID --mode non-interactive

CI/CD mode (best for automation workflows):

kubiya source sync SOURCE_ID --mode ci

Syncing with Git Operations

Force branch switch even with local changes:

kubiya source sync SOURCE_ID --force

Commit local changes during sync:

kubiya source sync SOURCE_ID --add --commit "Automatic commit by Kubiya sync"

Commit and push changes:

kubiya source sync SOURCE_ID --add --commit "Automatic commit" --push

Updating Source Configuration

To modify existing source settings:

kubiya source update SOURCE_ID --branch "develop"

You can update various parameters:

kubiya source update SOURCE_ID \
                   --branch "feature/new-tools" \
                   --auth-token "NEW_TOKEN" \
                   --runner "new-runner" \
                   --remote "upstream"

Enable or disable automatic syncing:

kubiya source update SOURCE_ID --disable-auto-sync
kubiya source update SOURCE_ID --enable-auto-sync

Deleting Sources

To remove a source from your organization:

kubiya source delete SOURCE_ID

You'll be prompted to confirm the deletion.

To bypass confirmation:

kubiya source delete SOURCE_ID --force

Debugging Sources

If you encounter issues with a source, you can use the debug command:

kubiya source debug SOURCE_ID

This provides detailed information about the source and can help identify configuration issues.

Debug Options

Get detailed information about tools:

kubiya source debug SOURCE_ID --tools

Check Git repository status:

kubiya source debug SOURCE_ID --git

Verify source configuration:

kubiya source debug SOURCE_ID --config

Show contents of specific tools:

kubiya source debug SOURCE_ID --tools --show-content

Fix common issues automatically:

kubiya source debug SOURCE_ID --fix

Git Integration Features

Kubiya CLI has deep Git integration for source management:

Git Repository Detection

The CLI can automatically detect Git repositories:

kubiya source scan ~/projects --git-only

Branch Management

Working with branches:

# Update a source to use a different branch
kubiya source update SOURCE_ID --branch "feature/new-feature"

# Sync a specific branch without changing the source configuration
kubiya source sync SOURCE_ID --branch "hotfix/quick-fix"

Working with Uncommitted Changes

When syncing sources with local changes:

# Check for uncommitted changes
kubiya source debug SOURCE_ID --git

# Stash changes, sync, then reapply
kubiya source sync SOURCE_ID --stash

# Commit changes as part of sync
kubiya source sync SOURCE_ID --add --commit "Auto-commit during sync"

# Push changes after sync
kubiya source sync SOURCE_ID --add --commit "Auto-commit" --push

Source Structure

A typical Kubiya source follows a specific structure:

my-source/
├── kubiya.yaml        # Source configuration
├── tools/
│   ├── tool1.yaml     # Tool definition
│   ├── tool2.yaml     # Tool definition
│   └── group/
│       └── tool3.yaml # Grouped tool definition
├── scripts/
│   ├── script1.py     # Tool implementation
│   └── script2.sh     # Tool implementation
└── docs/
    └── README.md      # Documentation

Source Configuration (kubiya.yaml)

The kubiya.yaml file at the root of your source defines its configuration:

name: "My Automation Source"
description: "Collection of tools for DevOps automation"
version: "1.0.0"
maintainers:
  - name: "Jane Doe"
    email: "[e-mail]"
tools:
  - path: "tools/tool1.yaml"
  - path: "tools/tool2.yaml"
  - path: "tools/group/tool3.yaml"

Tool Definition Example

A tool definition in YAML format:

name: "deploy-app"
description: "Deploy an application to the specified environment"
inputs:
  - name: "app_name"
    description: "The name of the application to deploy"
    type: "string"
    required: true
  - name: "environment"
    description: "Target environment (dev, staging, prod)"
    type: "string"
    enum: ["dev", "staging", "prod"]
    required: true
script: "../scripts/deploy_app.py"
documentation: "Deploys the specified application to the target environment."

Advanced Tool Features

The Kubiya CLI supports several advanced tool configurations:

Long-Running Tools

For tools that run continuously (like services):

name: "monitoring-service"
description: "Run a monitoring service"
script: "../scripts/monitor.py"
longRunning: true

Tools with Volumes

For tools that need persistent storage:

name: "data-processor"
description: "Process large datasets"
script: "../scripts/process.py"
withVolumes:
  - name: "data-volume"
    mountPath: "/data"

Tools with Environment Variables

name: "aws-tool"
description: "AWS utility"
script: "../scripts/aws-util.py"
environment:
  AWS_REGION: "us-west-2"
  DEBUG: "true"

Best Practices

1. Organize Tools Logically: Group related tools together in subdirectories.

2. Include Documentation: Add README files and documentation for your sources and tools.

3. Version Control: Keep your sources in Git repositories for version control.

4. Regular Syncing: Sync your sources regularly to ensure teammates have access to the latest tools.

5. CI/CD Integration: Include source syncing in your CI/CD pipelines.

6. Use Runner-Specific Sources: Create sources tailored to specific runners for better performance and separation of concerns.

7. Use Dynamic Configuration: Implement environment-specific configuration with variables.

End-to-End Example: Setting Up a Complete Source

Here's a workflow for creating and managing a complete automation source:

Step 1: Create a Git Repository

mkdir aws-automation
cd aws-automation
git init

Step 2: Create the Source Structure

mkdir -p tools scripts docs
touch kubiya.yaml

Step 3: Define the Source Configuration

Create kubiya.yaml:

name: "AWS Automation"
description: "Tools for AWS infrastructure management"
version: "1.0.0"
maintainers:
  - name: "Your Name"
    email: "[e-mail]"
tools:
  - path: "tools/s3-bucket-create.yaml"
  - path: "tools/ec2-instance-manage.yaml"

Step 4: Create Tool Definitions

Create tools/s3-bucket-create.yaml:

name: "s3-bucket-create"
description: "Create an S3 bucket with specified configuration"
inputs:
  - name: "bucket_name"
    description: "Name of the S3 bucket"
    type: "string"
    required: true
  - name: "region"
    description: "AWS region"
    type: "string"
    default: "us-west-2"
script: "../scripts/s3_bucket_create.py"

Step 5: Implement the Tool

Create scripts/s3_bucket_create.py:

#!/usr/bin/env python3
import boto3
import sys
import json

def create_bucket(bucket_name, region):
    try:
        s3_client = boto3.client('s3', region_name=region)
        s3_client.create_bucket(
            Bucket=bucket_name,
            CreateBucketConfiguration={'LocationConstraint': region}
        )
        return True, f"Bucket {bucket_name} created successfully in {region}"
    except Exception as e:
        return False, str(e)

def main():
    # Parse inputs
    inputs = json.loads(sys.argv[1])
    bucket_name = inputs["bucket_name"]
    region = inputs.get("region", "us-west-2")
    
    # Create the bucket
    success, message = create_bucket(bucket_name, region)
    
    # Return result
    result = {
        "success": success,
        "message": message
    }
    print(json.dumps(result))

if __name__ == "__main__":
    main()

Step 6: Add Documentation

Create docs/README.md:

# AWS Automation

This source contains tools for AWS infrastructure management.

## Tools

### s3-bucket-create
Creates an S3 bucket with the specified configuration.

#### Usage:
```bash
kubiya tool execute s3-bucket-create --args '{"bucket_name":"my-bucket","region":"us-west-2"}'

### Step 7: Commit and Push to GitHub

```bash
git add .
git commit -m "Initial commit of AWS automation tools"
git remote add origin https://github.com/your-org/aws-automation.git
git push -u origin main

Step 8: Add the Source to Kubiya

kubiya source add --name "aws-automation" --url "https://github.com/your-org/aws-automation.git"

Step 9: Verify the Source

# List sources to get the ID
SOURCE_ID=$(kubiya source list --output json | jq -r '.[] | select(.name=="aws-automation") | .id')

# View source details
kubiya source describe $SOURCE_ID

# Debug the source to check for any issues
kubiya source debug $SOURCE_ID

# List tools in the source
kubiya tool list --source $SOURCE_ID

Step 10: Execute a Tool from the Source

kubiya tool execute s3-bucket-create --args '{"bucket_name":"my-test-bucket","region":"us-west-2"}'

Step 11: Set Up Automated Syncing in CI/CD

In your CI/CD pipeline:

# Example in a GitHub Actions workflow
kubiya source sync $SOURCE_ID --mode ci

Or in a cron job:

# Script to sync sources
#!/bin/bash
export KUBIYA_API_KEY="your-api-key"
kubiya source sync --all --mode ci

For more information on source management, visit the official documentation.