Skip to content

CKS: Scaling of Kubernetes Cluster fails if there is external worker node is added #12633

@kiranchavala

Description

@kiranchavala

problem

CKS: Scaling of Kubernetes Cluster fails if there is external worker node is added

versions

ACS 4.22 , 4.22.1 -nightly

The steps to reproduce the bug

  1. Create a cks cluster with size 1 for worker nodes

  2. Add an external node to the cks cluster

https://docs.cloudstack.apache.org/en/4.22.0.0/plugins/cloudstack-kubernetes-service.html#add-an-external-vm-instance-as-a-worker-node-to-a-kubernetes-cluster

  1. Scale the cks cluster to size 3 ( worker nodes)

  2. Cks cluster scaling fails and cluster goes into alert state

Image

logs

2026-02-12 09:59:04,050 DEBUG [c.c.u.d.T.Transaction] (API-Job-Executor-46:[ctx-efb09e57, job-241, ctx-3d8bf3ff, ctx-7b1fe543]) (logid:76986184) Rolling back the transaction: Time = 7 Name =  API-Job-Executor-46; called by -TransactionLegacy.rollback:905-TransactionLegacy.removeUpTo:848-TransactionLegacy.close:672-Transaction.execute:36-FirewallManagerImpl.createFirewallRule:255-FirewallManagerImpl.createIngressFirewallRule:207-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:77-DelegatingMethodAccessorImpl.invoke:43-Method.invoke:569-AopUtils.invokeJoinpointUsingReflection:344-ReflectiveMethodInvocation.invokeJoinpoint:198
2026-02-12 09:59:04,063 ERROR [c.c.k.c.a.KubernetesClusterScaleWorker] (API-Job-Executor-46:[ctx-efb09e57, job-241, ctx-3d8bf3ff]) (logid:76986184) Scaling failed for Kubernetes cluster : test-cks, unable to update network rules com.cloud.exception.ManagementServerException: Failed to provision firewall rules for SSH access for the Kubernetes cluster : test-cks
	at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterActionWorker.addFirewallRulesForNodes(KubernetesClusterActionWorker.java:1037)
	at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterResourceModifierActionWorker.createFirewallRules(KubernetesClusterResourceModifierActionWorker.java:668)
	at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterResourceModifierActionWorker.setupKubernetesClusterIsolatedNetworkRules(KubernetesClusterResourceModifierActionWorker.java:699)
	at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterScaleWorker.scaleKubernetesClusterIsolatedNetworkRules(KubernetesClusterScaleWorker.java:151)
	at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterScaleWorker.scaleKubernetesClusterNetworkRules(KubernetesClusterScaleWorker.java:192)
	at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterScaleWorker.scaleUpKubernetesClusterSize(KubernetesClusterScaleWorker.java:473)
	at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterScaleWorker.scaleKubernetesClusterSize(KubernetesClusterScaleWorker.java:509)
	at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterScaleWorker.scaleCluster(KubernetesClusterScaleWorker.java:584)
	at com.cloud.kubernetes.cluster.KubernetesClusterManagerImpl.scaleKubernetesCluster(KubernetesClusterManagerImpl.java:2142)
	at jdk.internal.reflect.GeneratedMethodAccessor721.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:109)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
	at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
	at jdk.proxy3/jdk.proxy3.$Proxy537.scaleKubernetesCluster(Unknown Source)
	at org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd.execute(ScaleKu

What to do about it?

CKS cluster scaling should succeed if there is an external node added to the cluster

Metadata

Metadata

Type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions