Automate Cleanup of Vulnerable Images in ECR

Amazon Elastic Container Registry (ECR) is a fully-managed Docker container registry that simplifies storing and managing Docker images within the AWS environment for developers.

Old Images:

Before the introduction of automated tools like lifecycle policies, cleaning up your repository involved manually deleting outdated images or developing and running scripts to perform this task. Now, Lifecycle policies in Amazon ECR enable you to establish rules for automatically removing outdated container images. Additionally, you have the option to preview these rules to identify precisely which container images will be impacted when the rules are executed. This feature helps in maintaining organized repositories, simplifying the process of locating important code revisions, and reducing storage costs. Amazon ECR provides 2 ways using which we can delete the older images.

• Based on the image age
• Based on the image count

The following diagram shows the lifecycle policy workflow from AWS website:

Vulnerable Images:

ECR offers two image scanning options:

• Enhanced Image Scanning: This option integrates with Amazon Inspector for ongoing, automated scanning of your repositories, identifying vulnerabilities in both operating systems and programming language packages.

• Basic Image Scanning: Utilizes the Common Vulnerabilities and Exposures (CVEs) database from the open-source Clair project to detect vulnerabilities in images. Scans can be configured to run automatically when images are pushed or can be initiated manually.

While image scanning is a valuable feature for identifying vulnerabilities in your images, the responsibility to act on this information rests with you. By leveraging Amazon EventBridge, you can set up actions to respond to the outcomes of these scans, allowing for automated responses to detected vulnerabilities.

Automated Tagging Solution

When a scan is completed, ECR generates an event that includes the results of the vulnerability scan. This event can be captured by Amazon EventBridge. You can create an EventBridge rule that triggers an AWS Lambda function in response to the ECR image scan completion event. This Lambda function can then process the scan results.

One approach could be to automatically tag images that contain critical vulnerabilities with a specific prefix such as ‘vulnerable’. Images tagged in this manner could then be scheduled for removal from your repository after a certain number of days, depending on your specific requirements. Since a single tag cannot be assigned to multiple images within the same repository, utilizing prefixes enables you to categorize and manage images based on their vulnerability status effectively.

1. Create an Amazon EventBridge Create an Amazon EventBridge (formerly CloudWatch Events) rule. This rule is designed to trigger a Lambda function when an AWS Elastic Container Registry (ECR) image scan finds critical vulnerabilities.

2. Configure a LambdaFunction as target of the rule The Lambda function will use the put_image function of the ECR client to append the “vulnerable” tag prefix to the image. This method allows for tagging the image directly within ECR, eliminating the need to pull or push the Docker image again. Ensure your Lambda function has the necessary permissions to be invoked by EventBridge. This typically involves adding a permission to the Lambda function’s policy that allows the events.amazonaws.com service to invoke it.

3. Define Lifecycle Policies Based on Tag Status: You can create policies that target images with specific tags such as ‘vulnerable-xxxx’

Integrate ECR Scanning Results with CI/CD Pipeline

Implement a step in your CI/CD pipeline that queries the ECR for the scanning results of the base image before it’s used. You can use AWS CLI or SDKs to check the scan findings of the image. If vulnerabilities are found, you can fail the pipeline or send notifications to take action.

check_vulnerabilities:
  stage: prepare
  script:
    - aws ecr describe-image-scan-findings
     --repository-name name
     --image-id imageTag=tag_name
     --region us-east-2
    - # Script to analyze the findings and decide whether to proceed

Alternatively, you can focus on examining the tags, and if any tags with a vulnerable prefix are associated with the image, you can choose to fail the pipeline.

There are several third-party tools that can be integrated with GitLab CI/CD to scan your AWS ECR base images for vulnerabilities at the beginning of each job. These tools not only provide detailed vulnerability reports but can also be configured to break the build process if they find issues according to the severity levels you define. Here are some notable tools that can be used for this purpose:

1. Trivy
2. Anchore Engine
3. Sysdig Secure
4. Aqua Security Trivy (as part of Aqua Platform)