Integrating AWS ECR(Elastic Container Registry) with Kubernetes

Subscribe to my newsletter and never miss my upcoming articles

Integrating AWS ECR(Elastic Container Registry) with Kubernetes

I run my own kubernetes cluster spun up using Rancher on AWS lightsail which is an alternative to DigitalOcean. Though Lightsail is part of AWS, its not tightly as integrated as the rest of AWS. The ECR docker image token(or password) expires every 12 hours, and everytime you want to pull or push you have to renew it. To use it with kubernetes you need some way to update the secret automatically every 12 hours.

Getting ECR to work with it is like as same as any other non AWS(or EKS) cluster may take a bit of work . You may read further if you want to integrate it with your DIY or other non AWS kubernetes clusters.

I dockerized a lightweight python script to run as a k8s cron job, which will fetch a new login token every 6 hours(based on your deployment.yaml).

First create a secret that holds your AWS credentials with

kubectl create secret -n ecr-kube-helper generic ecr-kube-helper-ecr-secret --from-literal=REGION=[AWS_REGION] --from-literal=ID=[AWS_KEY_ID] --from-literal=SECRET=[AWS_SECRET]</span>

Lets begin by creating a service account.

apiVersion: v1
kind: Namespace
metadata:
  name: ecr-kube-helper
  labels:
    name: ecr-kube-helper
---
apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: ecr-kube-helper
  name: svac-ecr-kube-helper
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: role-ecr-kube-helper
  namespace: ecr-kube-helper
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get","delete", "create"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: ecr-kube-helper
  name: rb-ecr-kube-helper
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: role-ecr-kube-helper
subjects:
  - kind: ServiceAccount
    name: svac-ecr-kube-helper
    namespace: ecr-kube-helper

Save it as a yaml file and run the following command

kubectl apply -f ./service-account-ecr.yml

Then deploy it with a k8s CronJob deployment, but be sure to change the environment values.

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  namespace: ecr-kube-helper
  name: cron-ecr-kube-helper
  labels:
    app: cron-ecr-kube-helper
spec:
  schedule: "0 */6 * * *"
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 5
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          serviceAccountName: svac-ecr-kube-helper
          volumes:
            - name: svac-ecr-kube-helper-token-dr9bg
              secret:
                secretName: svac-ecr-kube-helper-token-dr9bg
          containers:
            - name: pod-ecr-kube-helper
              image: anaganisk/ecr-kube-helper:1.0.0
              imagePullPolicy: IfNotPresent
              volumeMounts:
                - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
                  name: svac-ecr-kube-helper-token-dr9bg
              env:
                - name: AWS_DEFAULT_REGION
                  valueFrom:
                    secretKeyRef:
                      # AWS credientials secret
                      name: ecr-kube-helper-ecr-secret
                      key: REGION
                - name: AWS_ACCESS_KEY_ID
                  valueFrom:
                    secretKeyRef:
                      # AWS credientials secret
                      name: ecr-kube-helper-ecr-secret
                      key: ID
                - name: AWS_SECRET_ACCESS_KEY
                  valueFrom:
                    secretKeyRef:
                      # AWS credientials secret
                      name: ecr-kube-helper-ecr-secret
                      key: SECRET
                - name: LOGLEVEL
                  value: INFO
                - name: TARGET_SECRET
                  value: xxxSecretxxx
                - name: TARGET_ECR
                  value: "xxxECR_REPOxxx"
                - name: TARGET_NAMESPACE
                  value: "ecr-kube-helper"
                - name: TARGET_EMAIL
                  value: "docker@example.com"

run the following command to deploy

kubectl apply -f ./deployment.yml

The helper can only update one ECR ID for now so, if you want to use it with multiple ECR IDs you may have to create multiple cronjobs. One ECR ID may have multiple repositories

ECR_ID.dkr.ecr.ap-south-1.amazonaws.com/repository

Github Page https://anaganisk.github.io/ecr-kube-helper/

No Comments Yet