Kubernetes on OpenStack: Why Cluster API is the right approach
If you want to run Kubernetes on your own OpenStack infrastructure, you essentially have three options: OpenStack Magnum, a manual kubeadm setup, or Cluster API with the OpenStack provider CAPO. All three work technically. But only one actually delivers what you would expect from modern infrastructure management.
What is wrong with the other approaches
OpenStack Magnum has been around for years. The project has aged, development moves slowly, and anyone who has tried to debug a Magnum issue knows how opaque the system is under the hood. Cluster upgrades are cumbersome, flexibility is limited, and clean integration with GitOps workflows is barely possible.
Manual kubeadm setups are the other extreme. You get full control, but you pay for it with considerable operational overhead. Upgrades have to be planned and executed node by node. Anyone running three or four clusters ends up spending more time on cluster management than on actual work.
What Cluster API does differently
Cluster API is a CNCF project that manages Kubernetes clusters using the same mechanisms Kubernetes uses for everything else: declarative resources, controllers, reconciliation loops. You describe the desired state of a cluster in YAML and the system ensures reality matches that state.
CAPO is the provider that connects Cluster API to OpenStack. It uses the OpenStack APIs to provision machines, configure networking and manage the cluster lifecycle. Initial deployment, upgrades, scaling out: everything goes through the same API, everything is versionable and reproducible.
The result is that Kubernetes clusters on OpenStack can be managed exactly like other Kubernetes resources. You can put them in Git, reconcile them with Flux or ArgoCD, and you have a traceable history of every change.
Every OpenStack environment is a little different
CAPO is flexible, but it is not a black box that configures itself. Network topologies, flavor configurations, handling of floating IPs or security groups all vary from environment to environment. Anyone who copies Cluster API manifests straight from a tutorial will almost always run into something unexpected.
A well-structured cluster template helps keep the configuration readable and reproducible. Here is a minimal example for CAPO v1beta2 with three control plane nodes and three workers. The first block is the Secret containing the OpenStack credentials, which CAPO references via identityRef:
apiVersion: v1
kind: Secret
metadata:
name: my-cluster-cloud-config
namespace: default
stringData:
clouds.yaml: |
clouds:
openstack:
auth:
auth_url: https://keystone.example.com/v3
username: "my-user"
password: "my-password"
project_name: "my-project"
user_domain_name: "Default"
project_domain_name: "Default"
region_name: "RegionOne"
interface: "public"
identity_api_version: 3
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
name: my-cluster
namespace: default
spec:
clusterNetwork:
pods:
cidrBlocks: ["10.244.0.0/16"]
services:
cidrBlocks: ["10.96.0.0/12"]
serviceDomain: cluster.local
controlPlaneRef:
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlane
name: my-cluster-control-plane
infrastructureRef:
apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: OpenStackCluster
name: my-cluster
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: OpenStackCluster
metadata:
name: my-cluster
spec:
identityRef:
cloudName: openstack
name: my-cluster-cloud-config
apiServerLoadBalancer:
enabled: true
externalNetwork:
id: <external-network-id>
managedSecurityGroups:
allowAllInClusterTraffic: true
managedSubnets:
- cidr: "10.6.0.0/24"
dnsNameservers:
- "1.1.1.1"
controlPlaneAvailabilityZones:
- nova
---
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlane
metadata:
name: my-cluster-control-plane
spec:
replicas: 3
version: v1.35.3
machineTemplate:
infrastructureRef:
apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: OpenStackMachineTemplate
name: my-cluster-control-plane
kubeadmConfigSpec:
clusterConfiguration:
controllerManager:
extraArgs:
- name: cloud-provider
value: external
initConfiguration:
nodeRegistration:
name: "{{ local_hostname }}"
kubeletExtraArgs:
- name: cloud-provider
value: external
- name: provider-id
value: "openstack:///{{ instance_id }}"
joinConfiguration:
nodeRegistration:
name: "{{ local_hostname }}"
kubeletExtraArgs:
- name: cloud-provider
value: external
- name: provider-id
value: "openstack:///{{ instance_id }}"
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: OpenStackMachineTemplate
metadata:
name: my-cluster-control-plane
spec:
template:
spec:
flavor: SCS-2V-4
rootVolume:
sizeGiB: 50
type: ceph-ssd
image:
filter:
name: ubuntu-2404-kube-v1.35.3
sshKeyName: my-ssh-key
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
name: my-cluster-worker
spec:
clusterName: my-cluster
replicas: 3
selector:
matchLabels: {}
template:
spec:
clusterName: my-cluster
version: v1.35.3
failureDomain: nova
bootstrap:
configRef:
apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: KubeadmConfigTemplate
name: my-cluster-worker
infrastructureRef:
apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: OpenStackMachineTemplate
name: my-cluster-worker
---
apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: KubeadmConfigTemplate
metadata:
name: my-cluster-worker
spec:
template:
spec:
joinConfiguration:
nodeRegistration:
name: "{{ local_hostname }}"
kubeletExtraArgs:
- name: cloud-provider
value: external
- name: provider-id
value: "openstack:///{{ instance_id }}"
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: OpenStackMachineTemplate
metadata:
name: my-cluster-worker
spec:
template:
spec:
flavor: SCS-4V-8
rootVolume:
sizeGiB: 50
type: ceph-ssd
image:
filter:
name: ubuntu-2404-kube-v1.35.3
sshKeyName: my-ssh-keyA few fields are worth explaining:
rootVolume is required when the flavor does not include local storage. SCS flavors without a storage suffix like SCS-2V-4 or SCS-4V-8 are diskless — CAPO then provisions a Cinder volume as the boot disk. The volume type is environment-specific and needs to match the available Cinder backends.
provider-id in kubeletExtraArgs links each Kubernetes node to the corresponding OpenStack instance. Without this, the cloud controller manager cannot correctly identify nodes. The value is resolved automatically from the VM’s cloud-init context at runtime.
apiServerLoadBalancer and externalNetwork control the load balancer in front of the Kubernetes API server. The externalNetwork ID is the UUID of the external network in OpenStack through which the load balancer receives a floating IP.
controlPlaneAvailabilityZones and failureDomain define which OpenStack availability zones are used for control plane nodes and workers respectively.
Pivot: Moving the management cluster to the workload cluster
Cluster API is typically started on a temporary bootstrap cluster, often a local kind cluster. Once the actual workload cluster is running stably, the CAPI control plane can be moved there. This is called a pivot.
The command for this is clusterctl move. It transfers all CAPI objects from the bootstrap cluster to the target cluster:
# Install CAPI and CAPO on the target cluster
clusterctl init \
--kubeconfig=<target-cluster-kubeconfig> \
--infrastructure openstack
# Run the pivot
clusterctl move \
--to-kubeconfig=<target-cluster-kubeconfig>After the pivot, the workload cluster manages itself. The bootstrap cluster can then be decommissioned. What was previously a helper is no longer part of the infrastructure.
Kubeconfig management after the pivot
This is one of the most commonly overlooked challenges: the kubeconfig for the management cluster needs to remain valid long-term. kubeadm issues certificates with a default lifetime of one year. This also applies to the credentials embedded in the kubeconfig. Anyone who loses track of this will one day lose access to their own management cluster.
Two ways to solve this:
Option 1: Annual renewal with kubeadm certs renew all. Works reliably, but requires an automated process so that the renewal does not get forgotten.
Option 2: Service account with a permanently valid token. Since Kubernetes 1.24, service accounts no longer get automatic token secrets. You create the secret explicitly:
apiVersion: v1
kind: ServiceAccount
metadata:
name: capi-admin
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: capi-admin-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: capi-admin
namespace: kube-system
---
apiVersion: v1
kind: Secret
metadata:
name: capi-admin-token
namespace: kube-system
annotations:
kubernetes.io/service-account.name: capi-admin
type: kubernetes.io/service-account-tokenThe token from this secret does not expire as long as the secret and service account exist. It should be stored securely and used only for access to the management cluster. For extra safety: automate certificate renewal and keep the service account token as a more robust fallback.
Air-gapped deployments as a special case
One of the more interesting challenges recently was a deployment in an environment with no regular internet access. No direct image pulls from public registries, no external registry reachable, no straightforward path to the outside.
This is solvable, but it requires preparation. You need an internal registry, all required images must be mirrored beforehand, and containerd must be configured to pull exclusively from that internal source. CAPO lets you deploy this configuration directly to every node via KubeadmConfigTemplate:
apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: KubeadmConfigTemplate
metadata:
name: my-cluster-worker
spec:
template:
spec:
files:
- path: /etc/containerd/certs.d/docker.io/hosts.toml
owner: root:root
permissions: "0644"
content: |
server = "https://docker.io"
[host."https://registry-mirror.example.com"]
capabilities = ["pull", "resolve"]
- path: /etc/containerd/certs.d/registry.k8s.io/hosts.toml
owner: root:root
permissions: "0644"
content: |
server = "https://registry.k8s.io"
[host."https://registry-mirror.example.com"]
capabilities = ["pull", "resolve"]
- path: /etc/containerd/certs.d/ghcr.io/hosts.toml
owner: root:root
permissions: "0644"
content: |
server = "https://ghcr.io"
[host."https://registry-mirror.example.com"]
capabilities = ["pull", "resolve"]
- path: /etc/containerd/certs.d/quay.io/hosts.toml
owner: root:root
permissions: "0644"
content: |
server = "https://quay.io"
[host."https://registry-mirror.example.com"]
capabilities = ["pull", "resolve"]
joinConfiguration:
nodeRegistration:
name: "{{ local_hostname }}"
kubeletExtraArgs:
- name: cloud-provider
value: external
- name: provider-id
value: "openstack:///{{ instance_id }}"The same files array belongs in the KubeadmControlPlane block under kubeadmConfigSpec as well, so that control plane nodes receive the same mirror configuration. Anyone who does not plan for this from the start will hit a wall at the first node upgrade at the latest.
These kinds of environments are not unusual in public sector and security-sensitive contexts. They do not need an internet exception, they need a deployment that was designed to work without one from the beginning.
Who benefits
CAPO makes sense wherever you want to run Kubernetes on your own OpenStack infrastructure in a maintainable, reproducible way and retain control over your own stack. This applies to larger organisations with their own data centres, public authorities that depend on sovereign infrastructure, or companies that simply do not want to use a public cloud for data protection reasons.
What you need going in: solid Kubernetes knowledge and a good understanding of your own OpenStack environment. CAPO takes away a lot of the ongoing operational work, but the initial configuration requires knowing what you are doing. With the right preparation, that investment pays off quickly.
Running OpenStack and thinking about how to get Kubernetes set up properly on top of it? Feel free to get in touch.