Kubernetes Clusters

The clusters resource type provisions managed Kubernetes clusters. Teabar supports multiple Kubernetes distributions and cloud providers, giving you flexibility for different training and testing scenarios.

Basic Usage

spec:
  resources:
    clusters:
      - name: main
        provider: hetzner
        type: talos
        nodes:
          controlPlane: 1
          workers: 3

Schema Reference

clusters:
  - name: string              # Required: Cluster name
    provider: string          # Required: Cloud provider (hetzner, aws, azure)
    type: string              # Required: K8s distribution (talos, kubeadm, eks, aks, k3s)
    version: string           # Optional: Kubernetes version (default: latest stable)
    enabled: boolean          # Optional: Enable/disable this cluster (default: true)
    region: string            # Optional: Override default region
    nodes:                    # Required: Node configuration
      controlPlane: integer   # Number of control plane nodes
      workers: integer        # Number of worker nodes (supports templating)
    nodeSize: string          # Optional: Instance type for nodes
    controlPlaneSize: string  # Optional: Instance type for control plane (if different)
    networking:               # Optional: Cluster networking
      podCidr: string         # Pod network CIDR (default: 10.244.0.0/16)
      serviceCidr: string     # Service network CIDR (default: 10.96.0.0/12)
      cni: string             # CNI plugin (cilium, calico, flannel)
    features:                 # Optional: Cluster add-ons to install
      - string
    addons:                   # Optional: Detailed add-on configuration
      name:
        enabled: boolean
        config: object
    dependsOn:                # Optional: Resource dependencies
      - string

Kubernetes Distributions

Talos Linux

Talos is a secure, immutable, minimal Linux distribution designed for Kubernetes. It’s the recommended choice for most training environments.

clusters:
  - name: main
    provider: hetzner
    type: talos
    version: "1.29"
    nodes:
      controlPlane: 1
      workers: 3
    nodeSize: cx31

Advantages:

Immutable OS - no shell access, highly secure
Fast provisioning (2-3 minutes)
Minimal attack surface
Declarative configuration

Best for: Security-focused training, production-like environments

Kubeadm

Standard Kubernetes installation using kubeadm on Ubuntu/Debian VMs.

clusters:
  - name: main
    provider: hetzner
    type: kubeadm
    version: "1.29"
    nodes:
      controlPlane: 1
      workers: 3
    nodeSize: cx31

Advantages:

Standard Kubernetes experience
Shell access to nodes
Familiar to most administrators

Best for: Kubernetes administration training, debugging exercises

K3s

Lightweight Kubernetes distribution from Rancher.

clusters:
  - name: main
    provider: hetzner
    type: k3s
    version: "1.29"
    nodes:
      controlPlane: 1
      workers: 2
    nodeSize: cx21

Advantages:

Lower resource requirements
Faster startup
Single binary installation
Built-in local storage

Best for: Resource-constrained environments, edge computing training

Amazon EKS

Managed Kubernetes on AWS.

clusters:
  - name: main
    provider: aws
    type: eks
    version: "1.29"
    region: us-east-1
    nodes:
      workers: 3
    nodeSize: t3.medium
    nodePools:
      - name: default
        instanceType: t3.medium
        minSize: 1
        maxSize: 5
        desiredSize: 3

Advantages:

AWS-managed control plane
Deep AWS service integration
Managed node groups with auto-scaling

Best for: AWS-focused training, EKS-specific features

Azure AKS

Managed Kubernetes on Azure.

clusters:
  - name: main
    provider: azure
    type: aks
    version: "1.29"
    region: eastus
    nodes:
      workers: 3
    nodeSize: Standard_B2s
    resourceGroup: teabar-training

Advantages:

Azure-managed control plane
Azure AD integration
Virtual nodes support

Best for: Azure-focused training, AKS-specific features

Node Configuration

Dynamic Worker Counts

Scale workers based on participant count:

spec:
  variables:
    - name: participant_count
      type: integer
      default: 10

  resources:
    clusters:
      - name: main
        provider: hetzner
        type: talos
        nodes:
          controlPlane: 1
          # 1 worker per 5 participants, minimum 2
          workers: "{{ max 2 (div .Variables.participant_count 5) }}"

Different Node Sizes

Use different instance types for control plane and workers:

clusters:
  - name: main
    provider: hetzner
    type: talos
    nodes:
      controlPlane: 3
      workers: 5
    controlPlaneSize: cx31    # 2 vCPU, 8GB for control plane
    nodeSize: cx21            # 2 vCPU, 4GB for workers

High Availability

For production-like environments, use multiple control plane nodes:

clusters:
  - name: production-like
    provider: hetzner
    type: talos
    nodes:
      controlPlane: 3         # HA control plane
      workers: 5
    controlPlaneSize: cx31
    nodeSize: cx41

Note

High availability requires an odd number of control plane nodes (1, 3, or 5) for etcd quorum.

Networking

CNI Plugins

Choose your Container Network Interface plugin:

clusters:
  - name: main
    provider: hetzner
    type: talos
    networking:
      cni: cilium
      podCidr: 10.244.0.0/16
      serviceCidr: 10.96.0.0/12

Cilium provides advanced features:

eBPF-based networking
Network policies
Service mesh capabilities
Hubble observability

Custom CIDR Ranges

Avoid conflicts with your network:

clusters:
  - name: main
    provider: hetzner
    type: talos
    networking:
      podCidr: 172.16.0.0/16      # Custom pod network
      serviceCidr: 172.17.0.0/16   # Custom service network

Cluster Features

Built-in Add-ons

Enable common cluster add-ons:

clusters:
  - name: main
    provider: hetzner
    type: talos
    nodes:
      controlPlane: 1
      workers: 3
    features:
      - ingress-nginx        # NGINX Ingress Controller
      - cert-manager         # TLS certificate management
      - metrics-server       # Resource metrics API
      - dashboard            # Kubernetes Dashboard

Detailed Add-on Configuration

For more control, use the addons section:

clusters:
  - name: main
    provider: hetzner
    type: talos
    addons:
      ingress-nginx:
        enabled: true
        config:
          controller:
            service:
              type: LoadBalancer
            metrics:
              enabled: true
      
      cert-manager:
        enabled: true
        config:
          installCRDs: true
      
      metrics-server:
        enabled: true
      
      dashboard:
        enabled: true
        config:
          service:
            type: ClusterIP

Available Features

Feature	Description
`ingress-nginx`	NGINX Ingress Controller for HTTP routing
`cert-manager`	Automatic TLS certificate management
`metrics-server`	Resource metrics for HPA and kubectl top
`dashboard`	Kubernetes Dashboard web UI
`prometheus`	Prometheus monitoring stack
`argocd`	ArgoCD GitOps controller
`external-dns`	Automatic DNS record management
`local-path-provisioner`	Local storage provisioner

Cluster Outputs

Reference cluster outputs in other resources:

spec:
  resources:
    clusters:
      - name: main
        provider: hetzner
        type: talos
        features:
          - ingress-nginx

    dns:
      - name: app
        type: A
        target: "{{ .Resources.clusters.main.ingress_ip }}"

    manifests:
      - name: kubeconfig-secret
        cluster: main
        template: |
          apiVersion: v1
          kind: Secret
          metadata:
            name: cluster-kubeconfig
          stringData:
            kubeconfig: |
              {{ .Resources.clusters.main.kubeconfig | indent 14 }}

Available cluster outputs:

Output	Description
`.kubeconfig`	Full kubeconfig for cluster access
`.api_server`	API server URL
`.ingress_ip`	Ingress controller load balancer IP
`.control_plane_ips`	List of control plane node IPs
`.worker_ips`	List of worker node IPs
`.cluster_ca`	Cluster CA certificate
`.id`	Cluster identifier

Multiple Clusters

Deploy multiple clusters for advanced scenarios:

spec:
  resources:
    clusters:
      # Shared infrastructure cluster
      - name: shared
        provider: hetzner
        type: talos
        nodes:
          controlPlane: 1
          workers: 2
        features:
          - ingress-nginx
          - cert-manager

      # Per-participant clusters
      - name: "participant-{{ .Index }}"
        count: "{{ .Variables.participant_count }}"
        provider: hetzner
        type: k3s
        nodes:
          controlPlane: 1
          workers: 1
        nodeSize: cx21

Warning

Per-participant clusters are resource-intensive. Consider using namespace isolation instead for most training scenarios.

Provider Comparison

Feature	Hetzner (Talos/K3s)	AWS (EKS)	Azure (AKS)
Cost	$	$$$	$$
Setup time	2-5 min	10-15 min	10-15 min
Control plane	Self-managed	AWS-managed	Azure-managed
Scaling	Manual	Auto-scaling	Auto-scaling
Regions	EU, US	Global	Global
Best for	Training, dev	AWS workloads	Azure workloads

Best Practices

Right-Size Your Clusters

Start with minimal resources:

# Development/training
clusters:
  - name: dev
    type: talos
    nodes:
      controlPlane: 1
      workers: 2
    nodeSize: cx21

# Production-like
clusters:
  - name: prod
    type: talos
    nodes:
      controlPlane: 3
      workers: 5
    controlPlaneSize: cx31
    nodeSize: cx41

Use Talos for Security Training

Talos’s immutable design makes it ideal for security-focused training:

clusters:
  - name: secure-cluster
    type: talos
    # No SSH, no shell, minimal attack surface
    nodes:
      controlPlane: 1
      workers: 3

Enable Metrics Early

Always include metrics-server for resource visibility:

clusters:
  - name: main
    features:
      - metrics-server    # Required for kubectl top
      - ingress-nginx     # Required for HTTP routing

Tag Clusters

Use labels for organization:

clusters:
  - name: main
    tags:
      environment: "{{ .Environment.Name }}"
      project: kubernetes-training
      cost-center: education

Related Resources

Helm Releases - Deploy apps to clusters
Kubernetes Manifests - Apply raw manifests
DNS - DNS for cluster services
Isolation Modes - Namespace vs cluster isolation