Writing a cluster document

A cluster document describes the cluster you want. It is the -f file of kubox admin create and kubox cluster create, and the configYAML of a Cluster resource. Start here, then use the full field list for everything else.

A key Kubox does not recognise is ignored, not reported. A typo is dropped silently. If a setting has no effect, check its spelling against the reference.

The smallest document that builds

cluster.yaml

metadata:
  clusterName: demo
  pulumi:
    projectName: demo

aws:
  region: ap-southeast-2
  nodeGroups:
    - vmType: t3.small
      count: 1
      role: control-plane
    - vmType: t3.medium
      count: 1
      role: worker

kubox admin create -f cluster.yaml

Four things are doing the work:

string

What the cluster is called.

string

Names the Pulumi project this cluster’s stacks belong to. It cannot change once the cluster is built.

string

default:"us-east-1"

Where it is built.

AwsNodeGroup[]

The machines. You need at least one control-plane group and usually one worker group.

Growing it

Each snippet below is a fragment to merge into the document above.

More workers, and the spot-instance trap

aws:
  nodeGroups:
    - vmType: t3.medium
      count: 3
      role: worker
      labels:
        nodetype: default
      spotInstance: false

spotInstance defaults to true. If no spot capacity is available, the request waits instead of failing — so the build sits with no error and no machine. Set spotInstance: false to use on-demand.

GPU nodes

GPUs need an AMI with the NVIDIA extensions, and a taint so ordinary pods do not land on them.

aws:
  nodeGroups:
    - vmType: g6.xlarge
      count: 1
      awsAMI: ami-05261ad044523904f
      osDiskSizeGB: 200
      role: worker
      systemExtensions:
        - siderolabs/nonfree-kmod-nvidia-lts
        - siderolabs/nvidia-container-toolkit-lts
      labels:
        nodetype: gpu
      taints:
        - key: gpu
          value: true
          effect: NoExecute

The AMI is region-specific. See using GPU instances for the images and instance types that have been tested.

Volumes that actually bind

aws:
  nodeGroups:
    - vmType: t3.medium
      count: 2
      role: worker
      awsIAMInstanceProfile: KuboxEC2Instance

awsIAMInstanceProfile is the IAM role these nodes run as. Without it, Kubox cannot create disks and your PersistentVolumeClaims stay Pending. Which value you need depends on how you are building:

Building via	Set it to
A cloud connection	Leave it out — nodes get the connection’s node profile
`kubox admin create` from your machine	Name one, e.g. `KuboxEC2Instance`
Neither, deliberately	`none` launches nodes with no AWS role

A hostname and TLS

dns:
  rootDomain: example.com

aws:
  ingress:
    enabled: true

Kubox checks the DNS zone before it builds, so a mistake fails straight away instead of halfway through.

Controllers

Controllers are installed from a registry. aws-ebs-csi-driver, aws-s3-csi-driver and runtime-class are on by default.

controllers:
  - name: metrics-server
    config:
      image: registry.k8s.io/metrics-server/metrics-server:v0.7.2
  - name: headlamp
    enabled: false

Roles and bindings

rbac:
  - name: developer
    rules:
      - apiGroups: [""]
        resources: ["*"]
        verbs: ["*"]

Roles are applied to the cluster during creation. See RbacRole and RoleBinding.

Tagging for cost reporting

tags:
  - key: Environment
    value: dev

Pulling from a private ECR registry

Add the credential provider extension to every node group whose workloads pull from a private ECR registry.

aws:
  nodeGroups:
    - vmType: t3.medium
      count: 2
      role: worker
      systemExtensions:
        - siderolabs/ecr-credential-provider

The Kubox agent does not need this. It is pulled from a public registry.

Where the document goes

Direct

kubox admin create -f cluster.yaml builds from your machine with your own credentials, no management plane.

Through a plane

kubox cluster create -f cluster.yaml asks a management plane to build it.

As a resource

Inside a Cluster resource as spec.configYAML, applied with kubectl.

Use --plan with either command to see what would be built without building it.

Every field

The complete cluster document reference, generated from the types that parse it.

Plane documents

The plane.yaml that describes a management plane.

Worked examples

GPU instances, secrets, and other advanced configuration.

Build your first cluster

The end-to-end walkthrough, with cleanup.

API

CLI

Config

Writing a cluster document

The smallest document that builds

Growing it

More workers, and the spot-instance trap

GPU nodes

Volumes that actually bind

A hostname and TLS

Controllers

Roles and bindings

Tagging for cost reporting

Pulling from a private ECR registry

Where the document goes

Direct

Through a plane

As a resource

Next

Every field

Plane documents

Worked examples

Build your first cluster

​The smallest document that builds

​Growing it

​More workers, and the spot-instance trap

​GPU nodes

​Volumes that actually bind

​A hostname and TLS

​Controllers

​Roles and bindings

​Tagging for cost reporting

​Pulling from a private ECR registry

​Where the document goes

Direct

Through a plane

As a resource

​Next

Every field

Plane documents

Worked examples

Build your first cluster

The smallest document that builds

Growing it

More workers, and the spot-instance trap

GPU nodes

Volumes that actually bind

A hostname and TLS

Controllers

Roles and bindings

Tagging for cost reporting

Pulling from a private ECR registry

Where the document goes

Next