🏠 KCPilot / Quick Start
📝 Edit on GitHub

Quick Start

Get KCPilot running in under 5 minutes and perform your first Kafka cluster diagnostic.

Prerequisites

  • Rust toolchain (for building from source)
  • SSH access to your Kafka cluster environment
  • Basic familiarity with SSH configuration

Installation

# Clone the repository
git clone https://github.com/softwaremill/kcpilot.git
cd kcpilot

# Build the binary
cargo build --release

# Add to PATH (optional)
export PATH="$PATH:$(pwd)/target/release"

First Scan

Local Scan (from bastion host)

If you’re running directly on a machine that has access to your Kafka brokers:

kcpilot scan --broker kafka-poligon-dc1-1.c.sml-sandbox.internal:9092

where --broker is an address to one of your brokers

Remote Scan (through SSH bastion)

If you need to connect through a bastion host, first ensure your SSH config is set up:

# Add to ~/.ssh/config
Host kafka-prod
    HostName your-bastion.example.com
    User your-username
    ForwardAgent yes
    IdentityFile ~/.ssh/your-key

Then run the scan:

kcpilot scan --bastion kafka-prod --broker kafka-poligon-dc1-1.c.sml-sandbox.internal:9092

Understanding the Output

KCPilot creates a timestamped directory with all collected data:

kafka-scan-2024-01-15-14-30-45/
├── brokers/                    # Individual broker data
│   ├── broker_1/
│   │   ├── configs/           # Configuration files
│   │   ├── logs/              # Log files
│   │   ├── metrics/           # JVM and system metrics
│   │   └── system/            # System information
│   └── ...
├── cluster/                    # Cluster-wide information
├── metrics/                    # Prometheus metrics (if available)
├── COLLECTION_SUMMARY.md       # Human-readable summary
└── scan_metadata.json         # Scan metadata

Basic Analysis

View Collection Summary

cat ./kafka-scan-*/COLLECTION_SUMMARY.md

Run AI Analysis (if OpenAI API key configured)

# Set up AI analysis
export OPENAI_API_KEY=your_api_key_here

# Analyze the collected data
kcpilot analyze ./kafka-scan-2024-01-15-14-30-45 --report terminal

Possible report formats are: terminal, markdown or json

Test Specific Issues

# List available analysis tasks
kcpilot task list

# Test for specific configuration issues
kcpilot task test minimum_cpu_cores ./kafka-scan-2024-01-15-14-30-45
kcpilot task test in_transit_encryption ./kafka-scan-2024-01-15-14-30-45

Common Troubleshooting

SSH Connection Issues

# Test SSH connectivity
ssh kafka-prod "echo 'Connection successful'"

# Verify SSH agent has your key loaded
ssh-add -l

Permission Issues

Ensure your SSH user has appropriate permissions:

  • Read access to Kafka configuration directories
  • Access to log files (may require sudo)
  • Ability to run system commands like ps, df, free

Getting Help