I have a k8s cluster with several deployments. The workload variation is handled by
Horizontal Pod Autoscalers (HPA) and the deployments have different
Priority Classes to allow or prevent pod preemption. Some deployments have a
Pod Disruption Budget (PDB).
I have two similar deployments with the same Priority Class, when deployment 1 scales up (say, to 5 pods), deployment 2 is preempted and ends up at 0 pods which, as I understand it, violates its PDB. The preempted pods have the following PDB.
apiVersion: policy/v1beta1 kind: PodDisruptionBudget metadata: name: <APP NAME>-pdb spec: minAvailable: 1 selector: matchLabels: app: <APP NAME>
For me, the key element in this PDB is
As per the k8s PDB documentation:
PDBs cannot prevent involuntary disruptions from occurring, but they do count against the budget.
Pods which are deleted or unavailable due to a rolling upgrade to an application do count against the disruption budget, but workload resources (such as Deployment and StatefulSet) are not limited by PDBs when doing rolling upgrades.
The documentation lists involuntary disruption examples:
- a hardware failure of the physical machine backing the node
- cluster administrator deletes VM (instance) by mistake
- cloud provider or hypervisor failure makes VM disappear
- a kernel panic
- the node disappears from the cluster due to cluster network partition
- eviction of a pod due to the node being out-of-resources.
Deployment 1’s HPA is causing deployment 2 to scale to 0 because the cluster is out of resources. My question is, why is the k8s scheduler allowing deployment 1 to scale up to it’s target, killing all deployment 2 pods, instead of scaling deployment 1 to
target-1 and leaving one deployment 2 pod?
Is there something I’m missing in the deployment configuration?