Promoting Builds

Promoting builds is the process of upgrading a build to staging or production. For ui builds, this is done by copying s3 content into different buckets. For backend services, this is done by updating tags on ECR images so they are deployed in different environments. k9 provides a way to build scripts that must be manually run to promote a build.

k9 automatically create promotion stacks as part of promote service and promote ui so the following commands may not need to be run if they already exist.

UI

From the app directory containing app-config.yml run

k9 create promote-ui

This will create a CloudFormation stack containing an AWS Automation document, a Lambda Function, and a Lambda Role. To promote a version, execute the document with Version and TargetEnv parameters. (Systems Manager > Automation > Execute Automation > Owned by me > {{appName}}-ui-promotion > Execute). There is also a link to the document in the Outputs of the CloudFormation stack. Enter the build version, ex: 0.1.0-2 and pick either sat or prd as the target. This will call the {{appName}}-ui-promotion lambda. k9 defines the builds bucket (build source) and the sat and prd buckets in the lambda’s Environment Variables. If the bucket names were generated incorrectly or your bucket URLs are updated, they can be changed here.

Cross-account permissions

If your production bucket is in a different AWS account than your development and SAT buckets, then you will need to create another promote-ui stack in your production account. Then you should grant the LambdaRole permission to copy contents from your builds-bucket in the non-production account.

Manually add the following bucket policy to your builds bucket. (S3 > Buckets > -builds bucket > Permissions > Bucket Policy).

Replace PRD_LAMBDA_ROLE_ARN with the ARN of the role created by the promote-ui stack in the production account.

Replace BUILDS_BUCKET_ARN with the arn of the bucket you are applying this policy to. Note the /* after the ARN in the GetObject statement.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "PRD_LAMBDA_ROLE_ARN"
            },
            "Action": "s3:GetObject",
            "Resource": "BUILDS_BUCKET_ARN/*"
        },
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "PRD_LAMBDA_ROLE_ARN"
            },
            "Action": "s3:ListBucket",
            "Resource": "BUILDS_BUCKET_ARN"
        }
    ]
}

The lambda in the non-production will not have access to the production account bucket. Similarly, the production account lambda will not have access to the non-production lambda. This way, promotions must be run within the account being promoted to.

Service

From the app directory containing app-config.yml run

k9 create promote-service

This will create a CloudFormation stack containing an AWS Automation document, a Lambda Function, and a Lambda Role. To promote a version, execute the document with Version and TargetEnv parameters. (Systems Manager > Automation > Execute Automation > Owned by me > {{appName}}-service-promotion > Execute). There is also a link to the document in the Outputs of the CloudFormation stack. Enter the build version from the ECR ex: 0.0.1-2 and then pick the tag to apply to the that version. The lambda will apply the selected tag to the selected version.

Cross-account permissions

This lambda will only work for applying tags within one ECR. Cross-account promotion requires docker running to pull and push images, which is not currently possible through an AWS lambda. Images must be copied between accounts through some other method, either an ec2 instance in your production account, or using aws cli credentials locally.

Automatically Moving Images Across AWS Accounts

For security purposes, you do not want any IAM role outside of your production account to be able to reach into the production ECR. You also do not want to generate long term AWS api secret access tokens. For these reasons, it is best to have your production cluster handle copying images from nonprod ECR, into the production account ECR. k9 provides a command to set up a k8s cron job to do exactly this. Perform the following prerequisite steps:

  • Find the IAM role for your prd cluster nodes

    • (prd AWS account) IAM > Roles > {{clusterName}}

  • Attach the AmazonEC2ContainerRegistryFullAccess policy

    • Add permissions > Attach policies > AmazonEC2ContainerRegistryFullAccess

    • This will grant the worker nodes the required permissions to push and pull images.

    • This policy will need to be manually removed when deleting the cluster. The worker nodes stack will fail to delete if this policy is still attached.

  • Attach a policy to your nonprod ECR

    • (nonprod AWS account) > ECR > appName > Permissions (left sidebar) > Edit policy JSON

    • Paste the following policy

    {
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "PrdAccountAccess",
      "Effect": "Allow",
      "Principal": {
        "AWS": "ENTITY_ARN"
      },
      "Action": [
        "ecr:BatchCheckLayerAvailability",
        "ecr:BatchGetImage",
        "ecr:GetDownloadUrlForLayer",
        "ecr:DescribeImages"
      ]
    }
  ]
}

    • Replace ENTITY_ARN with the ARN of the prd cluster nodes

Perform the following from each app directory you have deployed a service for.

If a ecr-sync-values.yml file exists in the app directory, edit this file. Make sure to enter the clusterName you wish to apply the cronjob to, and the AWS account number for the nonprod ECR.

If a ecr-sync-values.yml file does not exist in the app directory, run

k9 create prd-configs

This will create prd-app-config.yml and ecr-sync-values.yml files. At this stage it is not necessary to edit the prd-app-config.yml file, but the ecr-sync-values.yml file must be edited with the information listed above.

Once the ecr-sync-values.yml file has been edited, run the following command to deploy the cron job.

k9 create ecr-sync

Now any image tagged with test in the non-production account will automatically be copied into the production ECR.

You may manually delete the cronjob to stop this process, or run k9 delete ecr-sync from the app directory.