Managing Environments

Learn how to effectively manage multiple environments (development, staging, production) using KubeZero's modular approach and GitOps principles.

Environment Strategy Overview

KubeZero supports multiple environment management strategies:

Virtual Cluster Environments: Lightweight isolation within a single cluster
Namespace-Based Environments: Traditional Kubernetes namespace separation
Cluster-Based Environments: Complete cluster isolation per environment
Hybrid Approaches: Combining multiple strategies

Virtual Cluster Environments

Virtual clusters provide the best balance of isolation and resource efficiency.

Creating Virtual Environments

# modules/vcluster-environment/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

helmCharts:
  - name: vcluster
    repo: https://charts.loft.sh
    version: 0.15.0
    namespace: vcluster-dev
    releaseName: dev-environment
    valuesFile: values-dev.yaml

resources:
  - namespace.yaml
  - rbac.yaml

# modules/vcluster-environment/values-dev.yaml
vcluster:
  image: rancher/k3s:v1.28.2-k3s1

storage:
  size: 5Gi
  className: fast-ssd

networking:
  advanced:
    clusterDomain: "cluster.local"
    fallbackHostCluster: false

rbac:
  clusterRole:
    create: true

nodeSelector:
  environment: development

tolerations:
  - key: "development"
    operator: "Equal"
    value: "true"
    effect: "NoSchedule"

resources:
  limits:
    cpu: 1
    memory: 2Gi
  requests:
    cpu: 500m
    memory: 1Gi

Environment-Specific Configurations

Create configurations for each environment:

# Directory structure
environments/
├── development/
│   ├── kustomization.yaml
│   ├── vcluster-config.yaml
│   └── app-overrides/
├── staging/
│   ├── kustomization.yaml
│   ├── vcluster-config.yaml
│   └── app-overrides/
└── production/
    ├── kustomization.yaml
    ├── vcluster-config.yaml
    └── app-overrides/

# environments/development/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: vcluster-dev

resources:
  - ../../modules/vcluster-environment
  - ../../stacks/application-stack/manifests

patches:
  - target:
      kind: VCluster
      name: dev-environment
    patch: |-
      - op: replace
        path: /spec/resources/limits/cpu
        value: "500m"
      - op: replace
        path: /spec/resources/limits/memory
        value: "1Gi"

  - target:
      kind: Deployment
      name: my-app
    patch: |-
      - op: replace
        path: /spec/replicas
        value: 1
      - op: add
        path: /spec/template/spec/containers/0/env/-
        value:
          name: ENVIRONMENT
          value: "development"

Accessing Virtual Clusters

# Install vCluster CLI
curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-amd64"
sudo install -c -m 0755 vcluster /usr/local/bin

# Connect to development virtual cluster
vcluster connect dev-environment -n vcluster-dev

# Use kubectl with the virtual cluster
kubectl get pods
kubectl apply -f my-app.yaml

# Disconnect
vcluster disconnect

Namespace-Based Environments

For simpler setups or when virtual clusters aren't needed.

Environment Namespaces

# stacks/multi-environment/base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ../../../modules/my-application

commonLabels:
  app.kubernetes.io/part-of: my-platform

# stacks/multi-environment/overlays/development/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: development

resources:
  - ../../base

namePrefix: dev-

patches:
  - target:
      kind: Deployment
      name: my-app
    patch: |-
      - op: replace
        path: /spec/replicas
        value: 1
      - op: replace
        path: /spec/template/spec/containers/0/resources/requests/cpu
        value: "100m"
      - op: replace
        path: /spec/template/spec/containers/0/resources/requests/memory
        value: "128Mi"

configMapGenerator:
  - name: app-config
    literals:
      - ENVIRONMENT=development
      - LOG_LEVEL=debug
      - FEATURE_FLAGS=experimental-features

# stacks/multi-environment/overlays/production/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: production

resources:
  - ../../base

namePrefix: prod-

patches:
  - target:
      kind: Deployment
      name: my-app
    patch: |-
      - op: replace
        path: /spec/replicas
        value: 5
      - op: add
        path: /spec/template/spec/containers/0/env/-
        value:
          name: ENVIRONMENT
          value: "production"
      - op: replace
        path: /spec/template/spec/containers/0/resources/requests/cpu
        value: "500m"
      - op: replace
        path: /spec/template/spec/containers/0/resources/requests/memory
        value: "512Mi"

configMapGenerator:
  - name: app-config
    literals:
      - ENVIRONMENT=production
      - LOG_LEVEL=info
      - FEATURE_FLAGS=stable-features

Cluster-Based Environments

For maximum isolation and compliance requirements.

Multi-Cluster Setup

# registry/clusters/management/_gitops.yaml
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: environment-clusters
  namespace: argocd
spec:
  generators:
  - list:
      elements:
      - cluster: development
        url: https://dev-cluster.example.com
        namespace: default
        values:
          replicas: 1
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
      - cluster: staging
        url: https://staging-cluster.example.com
        namespace: default
        values:
          replicas: 3
          resources:
            requests:
              cpu: 200m
              memory: 256Mi
      - cluster: production
        url: https://prod-cluster.example.com
        namespace: default
        values:
          replicas: 5
          resources:
            requests:
              cpu: 500m
              memory: 512Mi
  template:
    metadata:
      name: '{{cluster}}-applications'
    spec:
      project: multi-cluster
      source:
        repoURL: https://github.com/company/applications
        targetRevision: HEAD
        path: environments/{{cluster}}
      destination:
        server: '{{url}}'
        namespace: '{{namespace}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

Configuration Management

Environment-Specific Secrets

Use External Secrets Operator for environment-specific secrets:

# environments/development/secrets.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: app-secrets
  namespace: development
spec:
  refreshInterval: 15s
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: app-secrets
    creationPolicy: Owner
  data:
  - secretKey: database-url
    remoteRef:
      key: secret/development/myapp
      property: database_url
  - secretKey: api-key
    remoteRef:
      key: secret/development/myapp
      property: api_key

# environments/production/secrets.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: app-secrets
  namespace: production
spec:
  refreshInterval: 1h  # Less frequent refresh for production
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: app-secrets
    creationPolicy: Owner
  data:
  - secretKey: database-url
    remoteRef:
      key: secret/production/myapp
      property: database_url
  - secretKey: api-key
    remoteRef:
      key: secret/production/myapp
      property: api_key

Configuration Hierarchy

Implement a configuration hierarchy for different environments:

# config/base/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  # Base configuration
  app.yaml: |
    server:
      port: 8080
      timeout: 30s
    logging:
      format: json
    features:
      metrics: true
      health_checks: true

# config/development/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  # Development overrides
  app.yaml: |
    server:
      port: 8080
      timeout: 30s
    logging:
      level: debug
      format: json
    features:
      metrics: true
      health_checks: true
      debug_endpoints: true
      mock_external_services: true

# config/production/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  # Production configuration
  app.yaml: |
    server:
      port: 8080
      timeout: 10s
    logging:
      level: info
      format: json
    features:
      metrics: true
      health_checks: true
      debug_endpoints: false
      security_headers: true

GitOps Environment Promotion

Automated Promotion Pipeline

# .github/workflows/promote-environments.yaml
name: Environment Promotion
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  deploy-development:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    - name: Deploy to Development
      run: |
        # Update development environment
        kubectl apply -k environments/development/
    
  test-staging:
    needs: deploy-development
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    - name: Deploy to Staging
      run: |
        kubectl apply -k environments/staging/
    - name: Run Integration Tests
      run: |
        # Run tests against staging
        npm run test:integration

  deploy-production:
    needs: test-staging
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    environment: production
    steps:
    - uses: actions/checkout@v3
    - name: Deploy to Production
      run: |
        kubectl apply -k environments/production/

Manual Promotion with ArgoCD

# argocd/promotion-workflow.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: staging-promotion
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/company/applications
    targetRevision: HEAD
    path: environments/staging
  destination:
    server: https://kubernetes.default.svc
    namespace: staging
  syncPolicy:
    automated:
      prune: false  # Manual approval required
      selfHeal: false
    syncOptions:
    - CreateNamespace=true

Environment Monitoring

Environment-Specific Monitoring

# monitoring/environment-dashboards.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: environment-dashboard
data:
  dashboard.json: |
    {
      "dashboard": {
        "title": "Environment Overview",
        "panels": [
          {
            "title": "Application Health by Environment",
            "targets": [
              {
                "expr": "up{job=\"my-app\"} by (environment)",
                "legendFormat": "{{environment}}"
              }
            ]
          },
          {
            "title": "Request Rate by Environment",
            "targets": [
              {
                "expr": "rate(http_requests_total[5m]) by (environment)",
                "legendFormat": "{{environment}}"
              }
            ]
          }
        ]
      }
    }

Alerting Rules

# monitoring/alerts.yaml
groups:
- name: environment.rules
  rules:
  - alert: DevelopmentEnvironmentDown
    expr: up{job="my-app",environment="development"} == 0
    for: 5m
    labels:
      severity: warning
      environment: development
    annotations:
      summary: "Development environment is down"
      
  - alert: ProductionEnvironmentDown
    expr: up{job="my-app",environment="production"} == 0
    for: 1m
    labels:
      severity: critical
      environment: production
    annotations:
      summary: "Production environment is down"
      runbook_url: "https://wiki.company.com/runbooks/production-outage"

Environment Lifecycle Management

Creating New Environments

Script to create new environments:

#!/bin/bash
# create-environment.sh

ENVIRONMENT_NAME=$1
CLUSTER_TYPE=$2  # vcluster, namespace, or cluster

if [ -z "$ENVIRONMENT_NAME" ] || [ -z "$CLUSTER_TYPE" ]; then
    echo "Usage: $0 <environment-name> <cluster-type>"
    exit 1
fi

case $CLUSTER_TYPE in
    "vcluster")
        echo "Creating vCluster environment: $ENVIRONMENT_NAME"
        
        # Create vCluster configuration
        mkdir -p environments/$ENVIRONMENT_NAME
        cat > environments/$ENVIRONMENT_NAME/kustomization.yaml <<EOF
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: vcluster-$ENVIRONMENT_NAME

resources:
  - ../../modules/vcluster-environment
  - ../../stacks/application-stack/manifests

namePrefix: ${ENVIRONMENT_NAME}-

patches:
  - target:
      kind: VCluster
      name: environment
    patch: |-
      - op: replace
        path: /metadata/name
        value: $ENVIRONMENT_NAME
EOF
        ;;
        
    "namespace")
        echo "Creating namespace environment: $ENVIRONMENT_NAME"
        
        mkdir -p environments/$ENVIRONMENT_NAME
        cat > environments/$ENVIRONMENT_NAME/kustomization.yaml <<EOF
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: $ENVIRONMENT_NAME

resources:
  - namespace.yaml
  - ../../stacks/application-stack/manifests

namePrefix: ${ENVIRONMENT_NAME}-
EOF

        cat > environments/$ENVIRONMENT_NAME/namespace.yaml <<EOF
apiVersion: v1
kind: Namespace
metadata:
  name: $ENVIRONMENT_NAME
  labels:
    environment: $ENVIRONMENT_NAME
EOF
        ;;
        
    "cluster")
        echo "Creating cluster environment: $ENVIRONMENT_NAME"
        echo "Please provision cluster manually and update ArgoCD configuration"
        ;;
        
    *)
        echo "Invalid cluster type. Use: vcluster, namespace, or cluster"
        exit 1
        ;;
esac

# Create ArgoCD application
cat > argocd/applications/$ENVIRONMENT_NAME.yaml <<EOF
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: $ENVIRONMENT_NAME
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/company/applications
    targetRevision: HEAD
    path: environments/$ENVIRONMENT_NAME
  destination:
    server: https://kubernetes.default.svc
    namespace: $ENVIRONMENT_NAME
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true
EOF

echo "Environment $ENVIRONMENT_NAME created successfully!"
echo "Apply with: kubectl apply -f argocd/applications/$ENVIRONMENT_NAME.yaml"

Environment Cleanup

#!/bin/bash
# cleanup-environment.sh

ENVIRONMENT_NAME=$1

if [ -z "$ENVIRONMENT_NAME" ]; then
    echo "Usage: $0 <environment-name>"
    exit 1
fi

echo "Cleaning up environment: $ENVIRONMENT_NAME"

# Delete ArgoCD application
kubectl delete application $ENVIRONMENT_NAME -n argocd

# Delete environment resources
kubectl delete namespace $ENVIRONMENT_NAME
kubectl delete namespace vcluster-$ENVIRONMENT_NAME

# Remove configuration files
rm -rf environments/$ENVIRONMENT_NAME
rm -f argocd/applications/$ENVIRONMENT_NAME.yaml

echo "Environment $ENVIRONMENT_NAME cleaned up successfully!"

Best Practices

1. Environment Parity

Keep environments as similar as possible:

# Use the same base configuration
resources:
  - ../../stacks/application-stack/base

# Only patch differences
patches:
  - target:
      kind: Deployment
      name: my-app
    patch: |-
      - op: replace
        path: /spec/replicas
        value: 1  # Different per environment

2. Resource Management

Set appropriate resource limits per environment:

# Development - minimal resources
resources:
  requests:
    cpu: 50m
    memory: 64Mi
  limits:
    cpu: 200m
    memory: 256Mi

# Production - generous resources
resources:
  requests:
    cpu: 500m
    memory: 512Mi
  limits:
    cpu: 2000m
    memory: 2Gi

3. Security Isolation

Implement proper security boundaries:

# Network policies for environment isolation
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: environment-isolation
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          environment: development
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          environment: development

4. Configuration Management

Use a clear configuration hierarchy:

Base configuration: Common to all environments
Environment overlays: Environment-specific changes
Secrets: Always external, never in Git
Feature flags: Control features per environment

5. Promotion Strategy

Implement a clear promotion strategy:

Development → Staging → Production
     ↓            ↓          ↓
  Automatic   Semi-Auto   Manual

Troubleshooting Environments

Common Issues

Resource conflicts between environments

# Check resource usage
kubectl top nodes
kubectl top pods --all-namespaces

Configuration drift

# Compare configurations
kubectl diff -k environments/staging/
argocd app diff staging

Secret synchronization issues

# Check External Secrets Operator
kubectl get externalsecrets -A
kubectl describe externalsecret app-secrets -n staging

Debugging Commands

# Check environment status
kubectl get applications -n argocd | grep $ENVIRONMENT_NAME

# View environment logs
kubectl logs -l app=my-app -n $ENVIRONMENT_NAME

# Check vCluster status
vcluster list
kubectl get pods -n vcluster-$ENVIRONMENT_NAME

# Compare environment configurations
kubectl diff -k environments/development/
kubectl diff -k environments/production/

Next Steps

Security Guide - Secure your environments
Monitoring Guide - Monitor environment health
Backup and Restore - Protect your environments

Effective environment management is crucial for successful platform operations. Use these patterns and practices to create robust, scalable environment strategies with KubeZero.

Environment Strategy Overview​

Virtual Cluster Environments​

Creating Virtual Environments​

Environment-Specific Configurations​

Accessing Virtual Clusters​

Namespace-Based Environments​

Environment Namespaces​

Cluster-Based Environments​

Multi-Cluster Setup​

Configuration Management​

Environment-Specific Secrets​

Configuration Hierarchy​

GitOps Environment Promotion​

Automated Promotion Pipeline​

Manual Promotion with ArgoCD​

Environment Monitoring​

Environment-Specific Monitoring​

Alerting Rules​

Environment Lifecycle Management​

Creating New Environments​

Environment Cleanup​

Best Practices​

1. Environment Parity​

2. Resource Management​

3. Security Isolation​

4. Configuration Management​

5. Promotion Strategy​

Troubleshooting Environments​

Common Issues​

Debugging Commands​

Next Steps​