Self-Managed: cluster-autoscaler

Install cluster-autoscaler with Helm. Reference ArtifactHub for all possible options.

INFO

Cluster Autoscaler is restricted to only autoscale node groups created by Yellowbrick Operator.

When using the commands or values outlined here, please make appropriate substitutions defined as:

Value	Description
{cluster-name}	The name of the Kubernetes cluster
{cloud-provider}	Your cloud provider: aws, azure, or gce
{version}	The chart version of cluster-autoscaler
{image-repo}	The container image repository pushed by the Deployer
{namespace}	The Kubernetes namespace into which you want to install

Helm Chart

Running the Yellowbrick Deployer will push the Helm charts and container images you need into your cloud environment. For instructions on pushing assets using the Deployer, see the documentation.

Chart name: cluster-autoscaler

The get-assets subcommand can be used to find the version of chart name cluster-autoscaler, see cli reference.

Install Command

See Authenticating with ECR

helm install cluster-autoscaler oci://{image-repo}/cluster-autoscaler \
  -n {namespace}  \
  -f values.yaml \
  --version {version}

Values

Please note that you may need to adjust the node selector and toleration values as appropriate for your installation.

clusterName: &clusterName { cluster-name }

affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
            - key: app.kubernetes.io/name
              operator: In
              values:
                - cluster-autoscaler
        namespaceSelector: {}
        topologyKey: kubernetes.io/hostname

autoDiscovery:
  clusterName: *clusterName
  tags:
    - k8s.io/cluster-autoscaler/enabled
    - k8s.io/cluster-autoscaler/{cluster-name}
    - k8s.io/cluster-autoscaler/node-template/label/cluster.yellowbrick.io/owned

cloudProvider: { cloud-provider }

extraArgs:
  ignore-daemonsets-utilization: true
  logtostderr: true
  max-empty-bulk-delete: "10"
  max-graceful-termination-sec: "600"
  max-node-provision-time: 15m
  max-total-unready-percentage: "45"
  new-pod-scale-up-delay: 0s
  ok-total-unready-count: "3"
  scale-down-delay-after-add: 10m
  scale-down-delay-after-delete: 10s
  scale-down-delay-after-failure: 3m
  scale-down-unneeded-time: 10m
  scale-down-unready-time: 10m
  scale-down-utilization-threshold: "0.5"
  scan-interval: 10s
  skip-nodes-with-local-storage: "false"
  skip-nodes-with-system-pods: "false"
  stderrthreshold: info
  v: 4

extraVolumeMounts:
  - mountPath: /etc/ssl/certs/ca-certificates.crt
    name: ssl-certs
    readOnly: true

extraVolumes:
  - hostPath:
      path: /etc/ssl/certs/ca-bundle.crt
    name: ssl-certs

image:
  repository: { image-repo }/autoscaling/cluster-autoscaler
  tag: v1.29.4

podAnnotations:
  cluster-autoscaler.kubernetes.io/safe-to-evict: "false"

podDisruptionBudget: null

nodeSelector:
  cluster.yellowbrick.io/hardware_type: t3.large
  cluster.yellowbrick.io/node_type: yb-op-standard

tolerations:
  - effect: NoSchedule
    key: cluster.yellowbrick.io/owned
    operator: Equal
    value: "true"
rbac:
  serviceAccount:
    annotations:
      eks.amazonaws.com/role-arn: { role-arn }

Creating Cloud Infrastructure

AWS

When installing on AWS, an IRSA service account is used. For details on IRSA, please see the AWS documentation.

Create the IAM role:

aws iam create-role \
  --role-name yb-eks-pod-cluster-autoscaler-{instance-name}-{region} \
  --assume-role-policy-document file://trust-policy.json

The trust policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "{oidc-provider-arn}"
     },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "{oidc-provider}:sub": "system:serviceaccount:{namespace}:cluster-autoscaler-aws-cluster-autoscaler"
       }
     }
   }
  ]
}

The IAM policy:

aws iam put-role-policy \
  --role-name yb-eks-pod-cluster-autoscaler-{instance-name}-{region} \
  --policy-name diags-upload \
  --policy-document file://iam-policy.json

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "autoscaling:DescribeAutoScalingGroups",
        "autoscaling:DescribeAutoScalingInstances",
        "autoscaling:DescribeLaunchConfigurations",
        "autoscaling:DescribeScalingActivities",
        "ec2:DescribeImages",
        "ec2:DescribeInstanceTypes",
        "ec2:DescribeLaunchTemplateVersions",
        "ec2:GetInstanceTypesFromInstanceRequirements",
        "eks:DescribeNodegroup"
      ],
      "Resource": ["*"]
    },
    {
      "Effect": "Allow",
      "Action": [
        "autoscaling:SetDesiredCapacity",
        "autoscaling:TerminateInstanceInAutoScalingGroup"
      ],
      "Resource": ["*"]
    }
  ]
}

To the values above, add these values and include the ARN of the AWS IAM role in place of {role-arn}:

rbac:
  serviceAccount:
    annotations:
      eks.amazonaws.com/role-arn: { role-arn }