In our previous article we looked into KEDA and scalers it offers. We will discuss Azure Pipelines scaler offered to scale build agents. Kubernetes HorizontalPodScaler offers scale-out / in functionality based on memory or cpu utilization. Only few project’s CICD utilizes configured resource threshold for scale-out. If you have configured min and max pods to be 10 and 50 respectively, hardly new pods will be created. This would cause most of build jobs to be queued or even if there are no jobs 10 pods will be idle adding to your cloud bill.

KEDA’s Azure Pipeline scaler solves this by listening to agent pool for pending job requests. If a new build job is created and goes to pending state then an event would be triggered. Scaler configured in your cluster will listen to this event and spin new pod to serve the request.

Scaler configuration

Azure Pipeline scaler has three resources to be configured.

1. Secret – Create a PAT with just agent pool read access. Then create Kubernetes secret personalAccessToken variable with new PAT value. If you have a PAT for build agents do not use that as it might have additional permissions and secrets value can be decoded.
   

1. TriggerAuthentication – Identifies how scaler should authenticate for communicating with Azure Devops. It could read auth information from Secret or environment variables.
2. ScaledObject – Contains configuration that describes how to scale agents and authenticate. Scaling information and authentication needs to be configured in trigger sections. You will need to associate the resource that you want to scale, we will be associating with Azure build agents.

Example

apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: pipeline-auth
data:
  personalAccessToken: encodedpersonalaccesstoken
---
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: pipeline-trigger-auth
  namespace: default
spec:
  secretTargetRef:
    - parameter: personalAccessToken
      name: pipeline-auth
      key: personalAccessToken
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: azure-pipelines-scaledobject
  namespace: default
spec:
  scaleTargetRef:
    name: azdevops-deployment
  minReplicaCount: 1
  maxReplicaCount: 5 
  triggers:
  - type: azure-pipelines
    metadata:
      poolID: "1"
      organizationURLFromEnv: "AZP_URL"
    authenticationRef:
     name: pipeline-trigger-auth

The scaleTargetRef has reference to the deployment name that will be scaled. Agent pool id will be required and can be obtained using REST API. Organization URL will be obtained from the build agent pod environment variable.

After creating the resource you could check if scaler is authenticating and all configuration are correct by using below command

kubectl get scaledobject azure-pipelines-scaledobject

If column active and enabled contains value true then configuration is good and scaler is listening to events from Azure Devops agent pool.

Note: Azure Devops agent pool’s agent state will be shown as offline even after the pod is terminated. When scaler tries to retrieve pending job requests the response will include offline agent details. Request will timeout if response is huge is size and might cause scaler to fail scaling agents. To workaround this issue create a cronjob to remove offline agents periodically.

Have a minimum of one pod and desired maximum pods and observe how agents are scaled based on job requests.