Keeping Mutable Tags Like dev and latest Alive Under an ECR Lifecycle Policy

A Kubernetes Deployment in the development environment stopped starting, with ImagePullBackOff. When I looked at ECR, there was no image with the dev tag.

$ aws ecr describe-images --repository-name app --image-ids imageTag=dev
An error occurred (ImageNotFoundException) ... The image with imageId {imageTag:'dev'} does not exist

ECR tokens expire after 12 hours, so at first I suspected authentication, but the Secret had just been refreshed and another container in the same Pod could pull. It was not authentication. The image itself did not exist. What deleted it was the lifecycle policy.

Background: two kinds of tags on one image

In this repository, every build pushes two kinds of tags to one image.

dev and 1.12.370-dev from the same build point to the same digest. This becomes a problem later.

Cause: ECR expires images, not tags

The policy at the time of the incident had only two rules.

{"rules":[
  {"rulePriority":1,"description":"Keep 3 images with version tags",
   "selection":{"tagStatus":"tagged","tagPatternList":["*.*"],"countType":"imageCountMoreThan","countNumber":3},
   "action":{"type":"expire"}},
  {"rulePriority":2,"description":"Delete untagged images",
   "selection":{"tagStatus":"untagged","countType":"sinceImagePushed","countUnit":"days","countNumber":1},
   "action":{"type":"expire"}}
]}

*.* means "tags that contain a dot" and is a pattern aimed at version tags. dev has no dot, so it does not match. At first glance, dev is protected.

However, what ECR expires is not the tag but the image (digest). 1.12.370-dev matches *.*. If production builds run three times, 1.12.370-dev is pushed out of the "newest three", that image is expired, and the dev tag that was on the same image disappears with it.

The lower the build frequency of the development environment is compared with production, the more likely this is to happen. This time, production builds had run four times within a little less than a month after the last development build.

This behavior is written in the lifecycle policy evaluation rules. "An image is expired by exactly one or zero rules." "An image that matches the tagging requirements of a rule cannot be expired by a rule with a lower priority." This second half can be used for the fix.

Fix: protect mutable tags with a high-priority keep-1 rule

For each mutable tag, put a rule that "keeps only the newest one" at a higher priority than the rule for version tags.

{"rules":[
  {"rulePriority":1,"description":"Keep the newest latest",
   "selection":{"tagStatus":"tagged","tagPatternList":["latest"],"countType":"imageCountMoreThan","countNumber":1},
   "action":{"type":"expire"}},
  {"rulePriority":2,"description":"Keep the newest dev",
   "selection":{"tagStatus":"tagged","tagPatternList":["dev"],"countType":"imageCountMoreThan","countNumber":1},
   "action":{"type":"expire"}},
  {"rulePriority":3,"description":"Keep 3 generations of version tags",
   "selection":{"tagStatus":"tagged","tagPatternList":["*.*"],"countType":"imageCountMoreThan","countNumber":3},
   "action":{"type":"expire"}},
  {"rulePriority":4,"description":"Delete untagged after 1 day",
   "selection":{"tagStatus":"untagged","countType":"sinceImagePushed","countUnit":"days","countNumber":1},
   "action":{"type":"expire"}}
]}

imageCountMoreThan: 1 means "delete everything beyond one", so the newest one of the images with the mutable tag remains. When the mutable tag is moved to a new image, the old image has only its version tag and goes back under the generation management of rule 3. If it drops out of that too, it becomes untagged and is deleted by rule 4. The handover from rule to rule works well.

If you add environments and the number of mutable tags grows, add a keep-1 rule for each. If you forget to add one, that environment will eventually show the same symptom.

Dry-run with preview before applying

put-lifecycle-policy takes effect as soon as it is applied, and images that match the conditions are deleted within 24 hours. If you put a wrong policy, they are deleted before you have time to check. First pass the same JSON to start-lifecycle-policy-preview and look at the result.

policy=$(cat policy.json)
repo=app

aws ecr start-lifecycle-policy-preview \
  --repository-name "$repo" --lifecycle-policy-text "$policy"

# The preview is asynchronous. Wait for COMPLETE instead of a fixed sleep
until [ "$(aws ecr get-lifecycle-policy-preview --repository-name "$repo" \
           --query status --output text)" = COMPLETE ]; do sleep 5; done

# previewResults lists only the images that would be expired
aws ecr get-lifecycle-policy-preview --repository-name "$repo" \
  --query 'previewResults[*].{tags:imageTags,rule:appliedRulePriority}' --output json

aws ecr put-lifecycle-policy \
  --repository-name "$repo" --lifecycle-policy-text "$policy"

If an image that includes latest or dev appears in previewResults, the priority or the pattern is wrong.

Setting it by hand in the console is not reproducible, and you repeat the same mistake the next time you create a repository. If you make this preview → put sequence a shell script, embed the policy JSON as a heredoc, and put it in the repository, then when you want to change a rule you only have to edit it and run it again.

Something the preview revealed: protected images still count

The preview at the time of applying gave an unexpected result. With four version-tagged images, 1.12.359 / 1.12.365 / 1.12.368 (latest) / 1.12.370-dev (dev), 1.12.359 became a target for expiry.

If the dev image were excluded from the evaluation of rule 3, rule 3 would see three version tags and nothing should be deleted. In reality, the "newest three" were decided with the dev image included in the count, the dev image was protected and remained, and 1.12.359, one generation older, was pushed out by that amount.

"An image matched by a higher-priority rule is not expired by a lower-priority rule" is correct, but that does not mean it is "excluded from the count of the lower-priority rule". While the development image is newer than production, the number of production generations kept by version tag is one less than the configured value. It does not affect the protection of the mutable tags, so I left it as is, but if you want to secure the number of generations strictly, increase countNumber by the number of mutable tags.

The option I did not take: no version tag on dev

If you put only dev on development images and do not put 1.12.N-dev on them, there is no tag that matches *.*, so they are not deleted even with the old policy. The old image that the tag was moved away from becomes untagged and is deleted in one day. Only the workflow needs to change, and the policy does not have to be touched.

However, tracking by version tag is no longer possible. Other repositories were already configured to protect mutable tags with keep-1, so I matched those.

Deleted images do not come back

Even if you fix the policy, the deleted dev does not come back. You need to rebuild the development image and push it.

The order is to first restore the environment with a build and push, and then fix the policy. Fixing the policy first does not bring the environment back, and if you only do the build first and leave the policy alone, the image is deleted again once production builds have run three times.