Configuring CircleCI to publish a multi-architecture Docker image
This is the fifth article about my adventures trying to use my Apple M1 MacBook for development.
In the previous article, I described how to configure a CircleCI-based CI/CD pipeline to test a Docker image on both Intel and ARM platforms.
Currently, however, the pipeline doesn’t do anything with the image once the Arm-platform tests pass.
It leaves the image in Docker Hub with the tag test-build-${CIRCLE_SHA1?}.
In this article, I describe how to configure the pipeline to add a ‘proper’ tag (e.g. BUILD-*, x.y.z.RELEASE, etc) to the image without rebuilding it.
The other articles in this series are:
- Part 1 - My Apple M1 MacBook: lots of cores, memory and crashing containers
- Part 2 - Building multi-architecture Docker images for Intel and ARM
- Part 3 - Configuring a CircleCI-based pipeline to build multi-architecture Docker images
- Part 4 - Testing an Intel and Arm multi-architecture Docker image on CircleCI
- Part 6 - Developing the Eventuate Common library on an M1/Arm MacBook
- Part 7 - Configuring CircleCI to build, test and publish multi-architecture images for Eventuate Common
- Part 8 - Building a multi-architecture Docker image for Apache Kafka
- Part 9 - Publishing multi-architecture base images for services
- Part 10 - Publishing a multi-architecture Docker image for the Eventuate CDC service
- Part 11 - The Eventuate Tram Customers and Orders example now runs on Arm/M1 MacBook!!
Defining a deploy job
The pipeline’s build job is still running the original deploy-artifacts.sh script which pushes the single architecture image to Docker Hub before testing the multi-architecture image on Arm.
This no longer makes sense and so the first change is to define deploy job, which only runs deploy-artifacts.sh if the test-arm64 job succeeds:
...
deploy:
docker:
- image: cimg/base:stable
working_directory: ~/plantuml
steps:
- checkout
- setup_remote_docker:
version: 20.10.11
- run:
command: ./deploy-artifacts.sh
workflows:
version: 2.1
build-test-and-deploy:
jobs:
- build
- test-arm64:
requires:
- build
- deploy:
requires:
- test-arm64
The deploy job executes once the test-arm64 jobs succeeds.
Let’s now look at how to publish the multi-architecture with the correct tag.
Publishing the image with the desired tag
The original deploy-artifacts.sh script simply pushed the locally built image to Docker Hub that had a tag corresponding to the Git branch.
For example, a master branch build would push a BUILD-${CIRCLE_BUILD_NUM?} tag, a x.y.z.RELEASE branch build would push a x.y.z.RELEASE tag.
What’s different about building a multi-architecture image is that it’s already been pushed to Docker Hub with a test-build-${CIRCLE_SHA1?} tag.
I naively thought that I could assign the correct tag by simply executing the following sequence of commands:
docker pulldocker tagto add a desired tagdocker push
These commands successfully publish an appropriately tagged image. However, that image is an Intel-only image! Yet another reminder that a Docker daemon only supports a single architecture. Pulling a multi-architecture image only downloads an image for that architecture.
One solution would be to rebuild the multi-architecture image with the desired tag. But, one drawback of rebuilding the image is that it publishes an image that hasn’t been tested
After a lot of googling, I discovered that I could use the docker manifest command to ‘add’ the desired tag to the image.
As I described in Part 2 - Building multi-architecture Docker images for Intel and ARM, a manifest for a multi-architecture image is a JSON object that points to a collection of architecture-specific images.
For example:
% docker manifest inspect microservicesio/plantuml:test-build-11c53...
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
"manifests": [
{
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"size": 1994,
"digest": "sha256:...",
"platform": {
"architecture": "amd64",
"os": "linux"
}
},
{
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"size": 1994,
"digest": "sha256:28899af3cb9a1756149adef730ba0596d4ea334727d5a2f47d26746c066f20b3",
"platform": {
"architecture": "arm64",
"os": "linux"
}
}
]
}
This manifest, for example, specifies that the Arm-specific image is microservicesio/plantuml@sha256:28899af3cb9a1756149adef730ba0596d4ea334727d5a2f47d26746c066f20b3.
Adding the desired tag to an image is simply a matter of using docker manifest create to create a new manifest that references the previously created images and docker manifest push to push the manifest to Docker Hub.
Using docker buildx imagetools create to tag the image
I then stumbled across a simpler way to create and push a manifest using the docker buildx imagetools create command.
The docker buildx imagetools provides a set of commands for working on images in a registry.
For example, this command creates the specified multi-architecture image in Docker Hub from the previously pushed images.
docker buildx imagetools create -t microservicesio/plantuml:BUILD-999 \
microservicesio/plantuml@sha256:28899af3cb9a1756149adef730ba0596d4ea334727d5a2f47d26746c066f20b3 ...
Since it’s manipulating JSON metadata, the command is extremely fast.
The deploy-artifacts.sh script invokes docker buildx imagetools create with the source images obtained by using jq.
The jq commands extract the digests from the JSON manifest and turns them into image references:
SOURCES=$(docker manifest inspect docker.io/microservicesio/plantuml:${SRC_TAG} | \
jq -r '.manifests[].digest | sub("^"; "docker.io/microservicesio/plantuml@")')
docker buildx imagetools create -t ${TARGET_IMAGE} $SOURCES
Since none of these commands use the Docker Daemon, the deploy job does not need setup_remote_docker.
After making these changes, the master branch build published microservicesio/plantuml:BUILD-131 image.
I then created the 0.3.0.RELEASE branch.
Finally, I’m able to run PlantUML on my M1 MacBook!
Viewing the changes
To see the changes I made to the project, take a look at this Github commit.
Next steps
Configuring the CircleCI pipeline to publish a multi-architecture Docker image was great learning experience. It also enabled me to use PlantUML on my M1 Macbook! The next step is to apply the lessons I learned here to the Eventuate projects.
Premium content and office hours is now available for paid subscribers at 
