Kubernetes Pod Disruption Budgets not respected by Horizontal Pod Autoscaler

I have a k8s cluster with several deployments. The workload variation is handled by Horizontal Pod Autoscalers (HPA) and the deployments have different Priority Classes to allow or prevent pod preemption. Some deployments have a Pod Disruption Budget (PDB).

I have two similar deployments with the same Priority Class, when deployment 1 scales up (say, to 5 pods), deployment 2 is preempted and ends up at 0 pods which, as I understand it, violates its PDB. The preempted pods have the following PDB.

apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: <APP NAME>-pdb
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: <APP NAME>

For me, the key element in this PDB is minAvailable: 1.

As per the k8s PDB documentation:

PDBs cannot prevent involuntary disruptions from occurring, but they do count against the budget.
Pods which are deleted or unavailable due to a rolling upgrade to an application do count against the disruption budget, but workload resources (such as Deployment and StatefulSet) are not limited by PDBs when doing rolling upgrades.

The documentation lists involuntary disruption examples:

  • a hardware failure of the physical machine backing the node
  • cluster administrator deletes VM (instance) by mistake
  • cloud provider or hypervisor failure makes VM disappear
  • a kernel panic
  • the node disappears from the cluster due to cluster network partition
  • eviction of a pod due to the node being out-of-resources.

Deployment 1’s HPA is causing deployment 2 to scale to 0 because the cluster is out of resources. My question is, why is the k8s scheduler allowing deployment 1 to scale up to it’s target, killing all deployment 2 pods, instead of scaling deployment 1 to target-1 and leaving one deployment 2 pod?

Is there something I’m missing in the deployment configuration?

Related Posts

Comments

LEAVE A REPLY

Please enter your comment!
Please enter your name here

Stay Connected

21,935FansLike
2,507FollowersFollow
0SubscribersSubscribe

Recent Stories

Britons Already Worried Over Budgets as Cost-of-Living Crisis Gathers Speed – BloombergQuint

Britons Already Worried Over Budgets as Cost-of-Living Crisis Gathers Speed&nbsp

At many churches, pandemic hits collection plates, budgets – CT Post

Biltmore United Methodist Church of Asheville, North Carolina, is for sale.Already financially strapped because of shrinking membership and a struggling preschool, the congregation was...

Melton council's budgets wishlist | Brimbank & North West – Brimbank & North West | Star Weekly

The electrification of the Melton train line and upgrade of the Western Highway are among Melton council’s wishes in the state and federal budgets. The...

Performance Budgets (Budget.json)

Use a performance budget to assert thresholds for performance metrics. budget.json supports three types of budgets: Timing budgets: Assert thresholds for time-based performance metrics like...

Erin and Ben Napier of 'Home Town' Reveal the Best Upgrades for Small Budgets

Erin and Ben Napier of "Home Town" have done some massive makeovers, but these HGTV stars also know that not every homeowner can afford...