Configuring CircleCI to publish a multi-architecture Docker image

multi-architecture docker images   docker  

This is the fifth article about my adventures trying to use my Apple M1 MacBook for development. In the previous article, I described how to configure a CircleCI-based CI/CD pipeline to test a Docker image on both Intel and ARM platforms. Currently, however, the pipeline doesn’t do anything with the image once the Arm-platform tests pass. It leaves the image in Docker Hub with the tag test-build-${CIRCLE_SHA1?}. In this article, I describe how to configure the pipeline to add a ‘proper’ tag (e.g. BUILD-*, x.y.z.RELEASE, etc) to the image without rebuilding it.

The other articles in this series are:

Defining a deploy job

The pipeline’s build job is still running the original deploy-artifacts.sh script which pushes the single architecture image to Docker Hub before testing the multi-architecture image on Arm. This no longer makes sense and so the first change is to define deploy job, which only runs deploy-artifacts.sh if the test-arm64 job succeeds:

...
deploy:
  docker:
    - image: cimg/base:stable
  working_directory: ~/plantuml
  steps:
    - checkout
    - setup_remote_docker:
        version: 20.10.11
    - run:
        command: ./deploy-artifacts.sh
workflows:
version: 2.1
build-test-and-deploy:
  jobs:
    - build
    - test-arm64:
        requires:
          - build
    - deploy:
        requires:
          - test-arm64

The deploy job executes once the test-arm64 jobs succeeds. Let’s now look at how to publish the multi-architecture with the correct tag.

Publishing the image with the desired tag

The original deploy-artifacts.sh script simply pushed the locally built image to Docker Hub that had a tag corresponding to the Git branch. For example, a master branch build would push a BUILD-${CIRCLE_BUILD_NUM?} tag, a x.y.z.RELEASE branch build would push a x.y.z.RELEASE tag. What’s different about building a multi-architecture image is that it’s already been pushed to Docker Hub with a test-build-${CIRCLE_SHA1?} tag.

I naively thought that I could assign the correct tag by simply executing the following sequence of commands:

  1. docker pull
  2. docker tag to add a desired tag
  3. docker push

These commands successfully publish an appropriately tagged image. However, that image is an Intel-only image! Yet another reminder that a Docker daemon only supports a single architecture. Pulling a multi-architecture image only downloads an image for that architecture.

One solution would be to rebuild the multi-architecture image with the desired tag. But, one drawback of rebuilding the image is that it publishes an image that hasn’t been tested

After a lot of googling, I discovered that I could use the docker manifest command to ‘add’ the desired tag to the image. As I described in Part 2 - Building multi-architecture Docker images for Intel and ARM, a manifest for a multi-architecture image is a JSON object that points to a collection of architecture-specific images. For example:

% docker manifest inspect microservicesio/plantuml:test-build-11c53...
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1994,
         "digest": "sha256:...",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1994,
         "digest": "sha256:28899af3cb9a1756149adef730ba0596d4ea334727d5a2f47d26746c066f20b3",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      }
   ]
}

This manifest, for example, specifies that the Arm-specific image is microservicesio/plantuml@sha256:28899af3cb9a1756149adef730ba0596d4ea334727d5a2f47d26746c066f20b3.

Adding the desired tag to an image is simply a matter of using docker manifest create to create a new manifest that references the previously created images and docker manifest push to push the manifest to Docker Hub.

Using docker buildx imagetools create to tag the image

I then stumbled across a simpler way to create and push a manifest using the docker buildx imagetools create command. The docker buildx imagetools provides a set of commands for working on images in a registry. For example, this command creates the specified multi-architecture image in Docker Hub from the previously pushed images.

docker buildx imagetools create -t microservicesio/plantuml:BUILD-999 \
  microservicesio/plantuml@sha256:28899af3cb9a1756149adef730ba0596d4ea334727d5a2f47d26746c066f20b3 ...

Since it’s manipulating JSON metadata, the command is extremely fast.

The deploy-artifacts.sh script invokes docker buildx imagetools create with the source images obtained by using jq. The jq commands extract the digests from the JSON manifest and turns them into image references:

SOURCES=$(docker manifest inspect docker.io/microservicesio/plantuml:${SRC_TAG} | \
   jq -r '.manifests[].digest  | sub("^"; "docker.io/microservicesio/plantuml@")')

docker buildx imagetools create -t ${TARGET_IMAGE} $SOURCES

Since none of these commands use the Docker Daemon, the deploy job does not need setup_remote_docker.

After making these changes, the master branch build published microservicesio/plantuml:BUILD-131 image. I then created the 0.3.0.RELEASE branch. Finally, I’m able to run PlantUML on my M1 MacBook!

Viewing the changes

To see the changes I made to the project, take a look at this Github commit.

Next steps

Configuring the CircleCI pipeline to publish a multi-architecture Docker image was great learning experience. It also enabled me to use PlantUML on my M1 Macbook! The next step is to apply the lessons I learned here to the Eventuate projects.


multi-architecture docker images   docker  


Copyright © 2024 Chris Richardson • All rights reserved • Supported by Kong.

About Microservices.io

Microservices.io is brought to you by Chris Richardson. Experienced software architect, author of POJOs in Action, the creator of the original CloudFoundry.com, and the author of Microservices patterns.

Upcoming public workshops: Microservices and architecting for fast flow

In-person: Berlin and Milan

DevOps and Team topologies are vital for delivering the fast flow of changes that modern businesses need.

But they are insufficient. You also need an application architecture that supports fast, sustainable flow.

Learn more and register for one of my upcoming public workshops in November.

NEED HELP?

I help organizations improve agility and competitiveness through better software architecture.

Learn more about my consulting engagements, and training workshops.

LEARN about microservices

Chris offers numerous other resources for learning the microservice architecture.

Get the book: Microservices Patterns

Read Chris Richardson's book:

Example microservices applications

Want to see an example? Check out Chris Richardson's example applications. See code

Virtual bootcamp: Distributed data patterns in a microservice architecture

My virtual bootcamp, distributed data patterns in a microservice architecture, is now open for enrollment!

It covers the key distributed data management patterns including Saga, API Composition, and CQRS.

It consists of video lectures, code labs, and a weekly ask-me-anything video conference repeated in multiple timezones.

The regular price is $395/person but use coupon CCMHVSFB to sign up for $95 (valid until November 8th, 2024). There are deeper discounts for buying multiple seats.

Learn more

Learn how to create a service template and microservice chassis

Take a look at my Manning LiveProject that teaches you how to develop a service template and microservice chassis.

Signup for the newsletter


BUILD microservices

Ready to start using the microservice architecture?

Consulting services

Engage Chris to create a microservices adoption roadmap and help you define your microservice architecture,


The Eventuate platform

Use the Eventuate.io platform to tackle distributed data management challenges in your microservices architecture.

Eventuate is Chris's latest startup. It makes it easy to use the Saga pattern to manage transactions and the CQRS pattern to implement queries.


Join the microservices google group