Brainbox Helm Chart

Kubernetes deployment for Brainbox AI Gateway with local LLM and embeddings.

Quick Reference - Customer Deployment

What to Change	Where	When

What to Change	Where	When
Container version	`values.yaml` → `images.gateway.tag`	Upgrades
Database schema	`values.yaml` → `dbInit.schemaVersion`	Upgrades
Node name	`values-dedicated-node.yaml` → `dedicatedNode.nodeName`	Initial deploy
DB passwords	`values-dedicated-node.yaml` → `secrets.*`	Initial deploy
Client credentials	`values-dedicated-node.yaml` → `secrets.clientId/Secret`	Initial deploy
RunPod URL	`values-kind.yaml` → `gateway.externalLlm.host`	Kind testing

Edit the appropriate values file:

Production: Edit brainbox/values-dedicated-node.yaml
Kind/Local: Edit brainbox/values-kind.yaml

Architecture Overview

┌─────────────────────────────────────────────────────────────────┐
│                        Kubernetes Cluster                        │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────────┐  │
│  │   MSSQL     │  │  PGVector   │  │     Gateway API         │  │
│  │ (metadata)  │  │ (vectors)   │  │  (FastAPI + LangChain)  │  │
│  └──────┬──────┘  └──────┬──────┘  └───────────┬─────────────┘  │
│         │                │                     │                 │
│         └────────────────┼─────────────────────┘                 │
│                          │                                       │
│  ┌─────────────┐  ┌──────┴──────┐  ┌─────────────────────────┐  │
│  │ Doc Loader  │  │  Embedding  │  │    LLM (vLLM)           │  │
│  │  (FastAPI)  │  │  (GTE-Qwen) │  │  OSS-20B / OSS-120B     │  │
│  └─────────────┘  └─────────────┘  └─────────────────────────┘  │
│                          GPU Nodes                               │
└─────────────────────────────────────────────────────────────────┘

Services

Service	Type	Description	GPU

Service	Type	Description	GPU
mssql	StatefulSet	SQL Server for metadata, queues, settings	No
pgvector	StatefulSet	PostgreSQL with vector extension for embeddings	No
gateway-api	Deployment	Main API service (FastAPI + LangChain)	No
doc-loader	Deployment	Document processing service	No
embedding	Deployment	GTE-Qwen embedding model	1 GPU
llm	Deployment	vLLM serving OSS model	1+ GPUs

Prerequisites

Hardware Requirements

Component	Local Testing (20B)	Production (120B)

Component	Local Testing (20B)	Production (120B)
GPUs	0-2x NVIDIA (16GB+ VRAM)	2+ NVIDIA (80GB+ VRAM each)
RAM	16GB	256GB+
Storage	100GB SSD	500GB+ NVMe
CPU	8 cores	32+ cores

Software Requirements

Kubernetes 1.24+ (Kind for local, Rancher for production)
Helm 3.x
NVIDIA GPU Operator (for GPU nodes)
kubectl configured for your cluster

CLI Installation (Recommended)

The Brainbox CLI (bb) provides shortcuts for common Helm and Kubernetes operations.

# Navigate to the helm chart directory
cd /path/to/helm/brainbox

# Make the script executable
chmod +x brainbox-cli.sh

# Create symlinks for easy access
sudo ln -sf $(pwd)/brainbox-cli.sh /usr/local/bin/brainbox
sudo ln -sf $(pwd)/brainbox-cli.sh /usr/local/bin/bb

# Verify installation
bb --help

CLI Quick Reference

# Set environment (Kind or dedicated-node)
bb env kind                  # Switch to Kind configuration
bb env dedicated-node        # Switch to production configuration

# Deployment
bb install                   # Install Brainbox
bb upgrade                   # Upgrade existing deployment
bb uninstall                 # Remove Brainbox

# Monitoring
bb status                    # Quick pod status overview
bb status full               # Detailed status with events
bb watch                     # Live pod status updates
bb diag                      # Full diagnostics

# Logs
bb logs gateway              # Show gateway logs
bb logs gw -f                # Follow gateway logs in real-time

# Service management
bb restart gateway           # Restart a component
bb restart all               # Restart all deployments
bb port gateway              # Port-forward gateway to localhost:8080
bb expose gateway            # Expose gateway externally (0.0.0.0:8080)

# Database access
bb sql                       # Connect to MSSQL shell
bb psql                      # Connect to PostgreSQL shell

Production Deployment

For production deployments on a dedicated GPU server.

1. Configure Values

Edit brainbox/values-dedicated-node.yaml with your settings:

# Node configuration - REQUIRED
dedicatedNode:
  enabled: true
  nodeName: "your-node-name"  # Run: kubectl get nodes

# Secrets - REQUIRED (change these!)
secrets:
  mssqlSaPassword: "YourSecureMssqlPassword"
  postgresPassword: "YourSecurePostgresPassword"
  dbUserPassword: "YourSecureDbUserPassword"
  clientId: "your-client-id"
  clientSecret: "your-client-secret"

2. Install Brainbox

Using CLI (Recommended):

bb env dedicated-node
bb install

Using Helm directly:

helm install brainbox ./brainbox -f ./brainbox/values-dedicated-node.yaml \
  --wait --timeout 45m

3. Via Rancher UI

Rancher provides a GUI for managing Helm deployments with built-in values persistence.

Click Explore on your cluster (not Cluster Management)
Go to Apps → Charts in the left sidebar
Click Create → From YAML or upload the chart
Edit values in the YAML editor - these are saved by Rancher
Future upgrades via Apps → Installed Apps → Upgrade preserve your values

⚠️ Values Persistence Note:

✅ Values set via Rancher UI → Apps → Upgrade ARE persisted through upgrades
✅ Values set via helm upgrade -f values.yaml ARE persisted (the file is the source of truth)
❌ Direct ConfigMap edits (via Rancher Resources → ConfigMaps) are NOT persisted through Helm upgrades - Helm will overwrite them with values from the release

Using Rancher with Existing Helm Deployment

If you've already deployed via CLI, Rancher can still manage it:

Rancher auto-discovers Helm releases
Go to Apps → Installed Apps to see existing deployments
Changes made via helm upgrade are reflected in Rancher
Changes made via Rancher UI are standard Helm operations

4. Verify Deployment

Using CLI:

bb status
bb diag
bb logs gateway

Using kubectl:

# Check all pods are running
kubectl get pods -w

# Check database initialization
kubectl logs job/brainbox-db-init

# Check GPU allocation
kubectl describe nodes | grep -A5 "Allocated resources"

# View LLM logs (large model download on first start)
kubectl logs -f deployment/brainbox-llm-1

# Test API
kubectl port-forward svc/brainbox-gateway 8080:8080
curl http://localhost:8080/v1/alive

Local Development with Kind

For local testing. Kind creates a single-node Kubernetes cluster in Docker.

1. Install Prerequisites

# Install Kind
# Windows (Chocolatey)
choco install kind

# macOS
brew install kind

# Linux
curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.20.0/kind-linux-amd64
chmod +x ./kind && sudo mv ./kind /usr/local/bin/kind

# Install Helm
# Windows
choco install kubernetes-helm

# macOS
brew install helm

# Linux
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

2. Create Kind Cluster

# Creates a single-node cluster (control-plane only)
kind create cluster --config kind-config.yaml --name brainbox

3. Install NVIDIA GPU Support (if using local GPU)

# Install NVIDIA Container Toolkit
# See: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html

# For Kind with GPU, you may need additional configuration
# See: https://jacobtomlinson.dev/posts/2022/quick-hack-adding-gpu-support-to-kind/

4. Configure Values

Edit brainbox/values-kind.yaml with your RunPod URL:

gateway:
  model: "openai/gpt-oss-20b"
  aiModels: '[{"model": "openai/gpt-oss-20b", "servers": ["https://YOUR-RUNPOD-ID.runpod.io:443"], "activation_link": ""}]'
  externalLlm:
    enabled: true
    host: "YOUR-RUNPOD-ID.runpod.io"
    port: "443"

5. Deploy Brainbox

Using CLI (Recommended):

bb env kind
bb install

Using Helm directly:

helm install brainbox ./brainbox -f ./brainbox/values-kind.yaml --wait --timeout 45m

6. Verify and Access

Using CLI:

bb status
bb port gateway              # Forward to localhost:8080
# In another terminal:
curl http://localhost:8080/v1/alive

Using kubectl:

# Watch pods come up
kubectl get pods -w

# Check database initialization completed
kubectl logs job/brainbox-db-init

# Port-forward to test
kubectl port-forward svc/brainbox-gateway 8080:8080

# Test the API
curl http://localhost:8080/v1/alive

Note: The local Kind deployment uses external RunPod for LLM but still deploys embedding locally (requires 1 GPU). If you don't have a local GPU, you can also disable embedding by editing values-kind.yaml:

embedding:
  enabled: false

gateway:
  embedder: "runpod"

7. Cleanup

Using CLI:

bb uninstall
kind delete cluster --name brainbox

Using Helm:

helm uninstall brainbox
kind delete cluster --name brainbox

8. Install Rancher (Optional - for testing Rancher UI locally)

If you want to test with Rancher UI locally (as customers will have in production):

# Run the install script (installs Rancher + creates NodePort service)
./install-rancher.sh

# Access immediately (no port-forward needed - persists after terminal close):
# https://localhost:30443  (or https://<server-ip>:30443 for remote access)

# Default credentials: admin / admin

Note: The script creates a NodePort service on port 30443, which provides persistent access without needing kubectl port-forward. The access survives terminal/SSH disconnections.

To uninstall Rancher:

./uninstall-rancher.sh

Deployment Architecture

All deployments use a single-node topology where all workloads run on one dedicated server. This is configured via dedicatedNode.nodeName in values.yaml.

# values.yaml - Set this to your target node
dedicatedNode:
  enabled: true
  nodeName: "your-node-name"  # kubectl get nodes

Docker Hub Registry Authentication

The chart automatically creates a Kubernetes secret with Docker Hub credentials for pulling private images from the businessgpt organization. This is configured in values.yaml:

imageRegistry:
  createSecret: true  # Auto-creates registry secret

The credentials are hardcoded in the chart templates. For manual image pulls:

docker login -u businessgpt -p <accesskey>

Database Initialization

The chart includes a post-install Job that automatically initializes the MSSQL database schema:

Waits for MSSQL to be ready
Creates the BG_Gateway database and user
Creates all required tables with indexes
Inserts default settings

Configuration in values.yaml:

dbInit:
  enabled: true
  dbName: "BG_Gateway"
  dbUser: "BG_GatewayUser"
  schemaVersion: "1.31.0"

GPU Node Setup (if using local LLM/Embedding)

If running LLM or embedding locally (not using RunPod), ensure your node has:

# Install NVIDIA drivers
sudo apt-get install -y nvidia-driver-535

# Install NVIDIA Container Toolkit
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/libnvidia-container/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit

# Install NVIDIA GPU Operator (manages GPU resources in Kubernetes)
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm install gpu-operator nvidia/gpu-operator --namespace gpu-operator --create-namespace

LLM Configuration

Deployment Options

Environment	LLM Location	Model	Configuration

Environment	LLM Location	Model	Configuration
Production	Local (Kubernetes)	OSS 120B	`llm.enabled: true`
Local Testing	External (RunPod)	OSS 20B	`llm.enabled: false` + `externalLlm`

External LLM (RunPod)

For local testing without deploying a local LLM, use an external RunPod server:

# In values-kind.yaml or custom values
llm:
  enabled: false  # Don't deploy local LLM

gateway:
  model: "openai/gpt-oss-20b"
  aiModels: '[{"model": "openai/gpt-oss-20b", "servers": ["https://xxx-runpod.io:443"], "activation_link": ""}]'
  externalLlm:
    enabled: true
    host: "xxx-runpod.io"  # Your RunPod endpoint
    port: "443"

Updating RunPod URL

After deployment, you can update the RunPod URL:

Option 1: Edit Values File + Upgrade (Recommended)

# Edit values-kind.yaml with new URL, then:
bb upgrade
# Or with Helm:
helm upgrade brainbox ./brainbox -f ./brainbox/values-kind.yaml

Option 2: Rancher UI

Navigate to your cluster in Rancher
Go to Apps → Installed Apps
Find brainbox and click Upgrade
Edit the values YAML with the new RunPod URL
Click Upgrade - values are persisted for future upgrades

Option 3: kubectl (temporary - won't persist through Helm upgrades)

# Edit the ConfigMap directly
kubectl edit configmap brainbox-config

# Update these values:
#   AiModels: '[{"model": "openai/gpt-oss-20b", "servers": ["https://NEW-RUNPOD-URL:443"], "activation_link": ""}]'
#   LLM_HOST: "NEW-RUNPOD-URL"

# Restart gateway to pick up changes
kubectl rollout restart deployment/brainbox-gateway

⚠️ Warning: Direct ConfigMap edits will be overwritten on the next helm upgrade. Use Option 1 or 2 for persistent changes.

Storage Options

Option 1: Local PV (Dedicated Node)

For a dedicated node with local disk:

storage:
  useLocalPV: true
  localPVBasePath: "/data/brainbox"
dedicatedNode:
  enabled: true
  nodeName: "worker-gpu-01"

Option 2: External StorageClass (NetApp, Cloud)

storage:
  storageClassName: "netapp-trident"
  mssql:
    size: 50Gi
  pgvector:
    size: 50Gi
  gateway:
    size: 10Gi

Option 3: HostPath (Development Only)

storage:
  useHostPath: true
  hostPathBase: "/data/brainbox"

GPU Scheduling

gpu:
  nodeSelector:
    gpu: "true"
  tolerations:
    - key: "nvidia.com/gpu"
      operator: "Exists"
      effect: "NoSchedule"

Files

brainbox/
├── Chart.yaml                    # Chart metadata
├── brainbox-cli.sh               # CLI tool for Helm/K8s management
├── values.yaml                   # BASE VALUES (image tags, all defaults)
├── values-kind.yaml              # OVERLAY: Local Kind testing (external RunPod)
├── values-dedicated-node.yaml    # OVERLAY: Production on dedicated GPU node
└── templates/
    ├── _helpers.tpl              # Template helpers
    ├── configmap-secret.yaml     # Environment config + secrets
    ├── databases.yaml            # MSSQL + PGVector StatefulSets
    ├── db-init-job.yaml          # Database initialization Job
    ├── deployments.yaml          # Gateway, Doc-loader, Embedding, LLM
    ├── httproute.yaml            # Gateway API ingress
    ├── local-storage.yaml        # Local PV StorageClass
    └── registry-secret.yaml      # Docker Hub credentials

Values File Structure

Overlay files (values-*.yaml) only contain overrides - everything else is inherited from values.yaml. This means:

Image tags: Always edit in values.yaml → images.*.tag
Environment-specific settings: Edit in the appropriate overlay file
Secrets: Edit in the appropriate overlay file (never pass on command line)

┌─────────────────────────────────────────────────────────────────┐
│                      values.yaml (BASE)                         │
│  - Image tags (images.gateway.tag, etc.)                        │
│  - Default secrets (override in overlay file)                   │
│  - Full resource definitions                                    │
│  - All configuration options with defaults                      │
└─────────────────────────────────────────────────────────────────┘
                              ▲
                              │ inherits from
        ┌─────────────────────┼─────────────────────┐
        │                     │                     │
┌───────┴───────┐    ┌────────┴────────┐   ┌───────┴───────┐
│ values-kind   │    │ values-dedicated │   │ Rancher UI    │
│   .yaml       │    │   -node.yaml     │   │   values      │
│               │    │                  │   │               │
│ - hostPath    │    │ - localPV        │   │ - Stored in   │
│ - no GPU      │    │ - GPU enabled    │   │   Helm        │
│ - RunPod LLM  │    │ - local LLM      │   │   release     │
│ - secrets     │    │ - secrets        │   │               │
└───────────────┘    └──────────────────┘   └───────────────┘

Upgrade / Uninstall

Using CLI:

bb upgrade                       # Upgrade with current environment's values
bb uninstall                     # Uninstall Brainbox

Using Helm:

# Upgrade with new values
helm upgrade brainbox ./brainbox -f ./brainbox/values-dedicated-node.yaml

# Uninstall (preserves PVCs by default)
helm uninstall brainbox

# Full cleanup including data
kubectl delete pvc --all
kubectl delete namespace brainbox  # If using a namespace

Troubleshooting

Database Init Job Failed

Using CLI:

bb diag init
bb logs init

Using kubectl:

kubectl describe job brainbox-db-init
kubectl logs job/brainbox-db-init
kubectl logs statefulset/brainbox-mssql

Image Pull Errors

# Check registry secret was created
kubectl get secret brainbox-registry -o yaml

# Manually verify credentials
kubectl run test-pull --rm -it --image=businessgpt/gateway_image:1.31.0 \
  --overrides='{"spec":{"imagePullSecrets":[{"name":"brainbox-registry"}]}}' \
  -- echo "Pull successful"

GPU Not Detected

# Check GPU operator pods
kubectl get pods -n gpu-operator

# Check node GPU resources
kubectl describe node <gpu-node> | grep nvidia

# Check if GPU is allocated to pods
kubectl describe pod <llm-pod> | grep -A5 "Requests"

Re-run Database Init

Using CLI:

bb db-reset

Using kubectl:

kubectl delete job brainbox-db-init
helm upgrade brainbox ./brainbox -f ./brainbox/values-dedicated-node.yaml

Deploying on Kubernetes

Brainbox Helm Chart

Quick Reference - Customer Deployment

Architecture Overview

Services

Prerequisites

Hardware Requirements

Software Requirements

CLI Installation (Recommended)

CLI Quick Reference

Production Deployment

1. Configure Values

2. Install Brainbox

3. Via Rancher UI

Using Rancher with Existing Helm Deployment

4. Verify Deployment

Local Development with Kind

1. Install Prerequisites

2. Create Kind Cluster

3. Install NVIDIA GPU Support (if using local GPU)

4. Configure Values

5. Deploy Brainbox

6. Verify and Access

7. Cleanup

8. Install Rancher (Optional - for testing Rancher UI locally)

Deployment Architecture

Docker Hub Registry Authentication

Database Initialization

GPU Node Setup (if using local LLM/Embedding)

LLM Configuration

Deployment Options

External LLM (RunPod)

Updating RunPod URL

Storage Options

Option 1: Local PV (Dedicated Node)

Option 2: External StorageClass (NetApp, Cloud)

Option 3: HostPath (Development Only)

GPU Scheduling

Files

Values File Structure

Upgrade / Uninstall

Troubleshooting

Database Init Job Failed

Image Pull Errors

GPU Not Detected

Re-run Database Init