Configuring CircleCI to build, test and publish multi-architecture images for Eventuate Common
In the previous post, I described the changes I made to the Eventuate Common project to locally build and test the Java libraries with the multi-architecture MySQL8 and Zookeeper images. In this post, you will learn how I configured the CircleCI pipeline to build, test and publish these images. I describe a useful pattern for structuring a pipeline that builds and tests a multi-architecture Docker image:
- Build and test a multi-architecture image locally (using a registry container) on both Intel and Arm.
- Build a multi-architecture image and push it to Docker Hub with a
test-build-*tag - Test the multi-architecture
test-build-*image on both Intel and Arm. - Create a manifest in Docker Hub with the desired tag
The other articles in this series are:
- Part 1 - My Apple M1 MacBook: lots of cores, memory and crashing containers
- Part 2 - Building multi-architecture Docker images for Intel and ARM
- Part 3 - Configuring a CircleCI-based pipeline to build multi-architecture Docker images
- Part 4 - Testing an Intel and Arm multi-architecture Docker image on CircleCI
- Part 5 - Configuring CircleCI to publish a multi-architecture Docker image
- Part 6 - Developing the Eventuate Common library on an M1/Arm MacBook
- Part 8 - Building a multi-architecture Docker image for Apache Kafka
- Part 9 - Publishing multi-architecture base images for services
- Part 10 - Publishing a multi-architecture Docker image for the Eventuate CDC service
- Part 11 - The Eventuate Tram Customers and Orders example now runs on Arm/M1 MacBook!!
Add CircleCI jobs that build and test locally on Intel and ARM
The first change was to replace the original build-and-test-mysql8 with a pair of jobs - build-and-test-mysql8-intel to build-and-test-mysql8-arm - that locally build and test the multi-architecture Docker image on Intel and Arm.
The two jobs are almost identical.
The only difference is that the build-and-test-mysql8-arm job specifies the arm.medium resource class:
- build-and-test/build-and-test:
name: build-and-test-mysql8-intel
script: |
docker context create tls-env
docker buildx create tls-env --use
docker run --privileged --rm tonistiigi/binfmt --install arm64,arm
export DOCKER_HOST_NAME=$(hostname)
./build-and-test-all-mysql8-multi-arch-locally.sh
skip_multi_arch_env_vars: "true"
- build-and-test/build-and-test:
name: build-and-test-mysql8-arm
resource_class: arm.medium
script: |
docker context create tls-env
docker buildx create tls-env --use
docker run --privileged --rm tonistiigi/binfmt --install amd64
export DOCKER_HOST_NAME=$(hostname)
./build-and-test-all-mysql8-multi-arch-locally.sh
skip_multi_arch_env_vars: "true"
Both jobs configure the usual docker buildx builder.
What’s unusual is that they both run docker run --privileged --rm tonistiigi/binfmt --install to install QEMU emulators.
This command is required because without it the docker buildx command fails.
error: failed to solve: rpc error: code = Unknown desc = process "/dev/.buildkit_qemu_emulator /bin/sh -c mkdir -p /usr/local/zookeeper-conf /usr/local/zookeeper-data && yum install -y wget tar gzip && (wget -q -O - https://archive.apache.org/dist/zookeeper/zookeeper-3.5.6/apache-zookeeper-3.5.6-bin.tar.gz | tar -xzf - -C /usr/local)" did not complete successfully: exit code: 2
Building a multi-platform image executes the commands in the Dockerfile for each architeture.
It uses the QEMU emulator for the non-native architectures.
And, as I described in my original article, the emulation is not perfect.
The Zookeeper Dockerfile needed updated emulators.
Adding a CircleCI job that builds and pushes the test-build images
The second change was a new CircleCI job - build-multi-arch-images - that builds the multi-architecture MySQL8 and Zookeeper images and pushes them to Docker Hub with a test-build-${CIRCLE_SHA1?} tag.
jobs:
build-multi-arch-images:
docker:
- image: cimg/base:stable
working_directory: ~/eventuate-common
steps:
- checkout
- setup_remote_docker:
version: 20.10.11
- run:
name: docker buildx build
command: |
. set-multi-arch-image-env-vars.sh
docker context create tls-env
docker buildx create tls-env --use
docker run --privileged --rm tonistiigi/binfmt --install arm64,arm
./build-multi-arch-images.sh
This job runs after the completion of the local build-and-test-mysql8-* jobs.
It’s similar to the job I described in a previous article.
The only difference is that it runs docker run --privileged --rm tonistiigi/binfmt --install before running docker buildx build.
Defining CircleCI jobs that test the multi-architecture images
The next change was to define a pair of jobs that test the multi-architecture images that the previous job pushed to Docker Hub. Here’s the configuration of the pipeline:
- build-and-test/build-and-test:
name: test-mysql8-zookeeper-multi-arch-intel
script: ./build-and-test-all-mysql8-multi-arch.sh
requires:
- build-multi-arch-images
- build-and-test/build-and-test:
name: test-mysql8-zookeeper-multi-arch-arm
script: ./build-and-test-all-mysql8-multi-arch.sh
resource_class: arm.medium
requires:
- build-multi-arch-images
These jobs are identical except that one runs on Intel and other runs on Arm.
Configuring CircleCI to publish the multi-architecture Docker images with the desired tag
The final change adds a job that uses docker buildx imagetools to create a manifest in Docker Hub with the desired label.
Here’s the configuration of the pipeline:
- deploy-multi-arch:
context:
- publish
requires:
- test-mysql8-zookeeper-multi-arch-intel
- test-mysql8-zookeeper-multi-arch-arm
Viewing the changes
To see the changes I made to the project, take a look at this Github commit.
Next steps
The MySQL8 and Zookeeper Docker images aren’t the only Docker images used by the Eventuate Tram Customers and Orders example application. In later articles, I’ll describe how I changed other projects to also build multi-architecture images.
Premium content and office hours is now available for paid subscribers at 
