In the previous post, I described the changes I made to the Eventuate Common project to locally build and test the Java libraries with the multi-architecture MySQL8 and Zookeeper images. In this post, you will learn how I configured the CircleCI pipeline to build, test and publish these images. I describe a useful pattern for structuring a pipeline that builds and tests a multi-architecture Docker image:
test-build-*
tagtest-build-*
image on both Intel and Arm.The other articles in this series are:
The first change was to replace the original build-and-test-mysql8
with a pair of jobs - build-and-test-mysql8-intel
to build-and-test-mysql8-arm
- that locally build and test the multi-architecture Docker image on Intel and Arm.
The two jobs are almost identical.
The only difference is that the build-and-test-mysql8-arm
job specifies the arm.medium
resource class:
- build-and-test/build-and-test:
name: build-and-test-mysql8-intel
script: |
docker context create tls-env
docker buildx create tls-env --use
docker run --privileged --rm tonistiigi/binfmt --install arm64,arm
export DOCKER_HOST_NAME=$(hostname)
./build-and-test-all-mysql8-multi-arch-locally.sh
skip_multi_arch_env_vars: "true"
- build-and-test/build-and-test:
name: build-and-test-mysql8-arm
resource_class: arm.medium
script: |
docker context create tls-env
docker buildx create tls-env --use
docker run --privileged --rm tonistiigi/binfmt --install amd64
export DOCKER_HOST_NAME=$(hostname)
./build-and-test-all-mysql8-multi-arch-locally.sh
skip_multi_arch_env_vars: "true"
Both jobs configure the usual docker buildx
builder.
What’s unusual is that they both run docker run --privileged --rm tonistiigi/binfmt --install
to install QEMU emulators.
This command is required because without it the docker buildx
command fails.
error: failed to solve: rpc error: code = Unknown desc = process "/dev/.buildkit_qemu_emulator /bin/sh -c mkdir -p /usr/local/zookeeper-conf /usr/local/zookeeper-data && yum install -y wget tar gzip && (wget -q -O - https://archive.apache.org/dist/zookeeper/zookeeper-3.5.6/apache-zookeeper-3.5.6-bin.tar.gz | tar -xzf - -C /usr/local)" did not complete successfully: exit code: 2
Building a multi-platform image executes the commands in the Dockerfile for each architeture.
It uses the QEMU emulator for the non-native architectures.
And, as I described in my original article, the emulation is not perfect.
The Zookeeper Dockerfile
needed updated emulators.
test-build
imagesThe second change was a new CircleCI job - build-multi-arch-images
- that builds the multi-architecture MySQL8 and Zookeeper images and pushes them to Docker Hub with a test-build-${CIRCLE_SHA1?}
tag.
jobs:
build-multi-arch-images:
docker:
- image: cimg/base:stable
working_directory: ~/eventuate-common
steps:
- checkout
- setup_remote_docker:
version: 20.10.11
- run:
name: docker buildx build
command: |
. set-multi-arch-image-env-vars.sh
docker context create tls-env
docker buildx create tls-env --use
docker run --privileged --rm tonistiigi/binfmt --install arm64,arm
./build-multi-arch-images.sh
This job runs after the completion of the local build-and-test-mysql8-*
jobs.
It’s similar to the job I described in a previous article.
The only difference is that it runs docker run --privileged --rm tonistiigi/binfmt --install
before running docker buildx build
.
The next change was to define a pair of jobs that test the multi-architecture images that the previous job pushed to Docker Hub. Here’s the configuration of the pipeline:
- build-and-test/build-and-test:
name: test-mysql8-zookeeper-multi-arch-intel
script: ./build-and-test-all-mysql8-multi-arch.sh
requires:
- build-multi-arch-images
- build-and-test/build-and-test:
name: test-mysql8-zookeeper-multi-arch-arm
script: ./build-and-test-all-mysql8-multi-arch.sh
resource_class: arm.medium
requires:
- build-multi-arch-images
These jobs are identical except that one runs on Intel and other runs on Arm.
The final change adds a job that uses docker buildx imagetools
to create a manifest in Docker Hub with the desired label.
Here’s the configuration of the pipeline:
- deploy-multi-arch:
context:
- publish
requires:
- test-mysql8-zookeeper-multi-arch-intel
- test-mysql8-zookeeper-multi-arch-arm
To see the changes I made to the project, take a look at this Github commit.
The MySQL8 and Zookeeper Docker images aren’t the only Docker images used by the Eventuate Tram Customers and Orders example application. In later articles, I’ll describe how I changed other projects to also build multi-architecture images.
Microservices.io is brought to you by Chris Richardson. Experienced software architect, author of POJOs in Action, the creator of the original CloudFoundry.com, and the author of Microservices patterns.
Chris helps clients around the world adopt the microservice architecture through consulting engagements, and training workshops.
Chris teaches comprehensive workshops for architects and developers that will enable your organization use microservices effectively.
Avoid the pitfalls of adopting microservices and learn essential topics, such as service decomposition and design and how to refactor a monolith to microservices.
Learn moreChris offers numerous other resources for learning the microservice architecture.
Want to see an example? Check out Chris Richardson's example applications. See code
Got a specific microservice architecture-related question? For example:
Consider signing up for a two hour, highly focussed, consulting session.
My virtual bootcamp, distributed data patterns in a microservice architecture, is now open for enrollment!
It covers the key distributed data management patterns including Saga, API Composition, and CQRS.
It consists of video lectures, code labs, and a weekly ask-me-anything video conference repeated in multiple timezones.
The regular price is $395/person but use coupon MECNPWNR to sign up for $120 (valid until May 16th, 2023). There are deeper discounts for buying multiple seats.
Take a look at my Manning LiveProject that teaches you how to develop a service template and microservice chassis.
Engage Chris to create a microservices adoption roadmap and help you define your microservice architecture,
Use the Eventuate.io platform to tackle distributed data management challenges in your microservices architecture.
Eventuate is Chris's latest startup. It makes it easy to use the Saga pattern to manage transactions and the CQRS pattern to implement queries.
Engage Chris to conduct an architectural assessment.
Note: tagging is work-in-process
anti-patterns · application api · application architecture · architecting · architecture documentation · assemblage · beer · containers · dark energy and dark matter · deployment · design-time coupling · development · devops · docker · eventuate platform · glossary · hexagonal architecture · implementing commands · implementing queries · inter-service communication · kubernetes · loose coupling · microservice architecture · microservice chassis · microservices adoption · microservicesio updates · multi-architecture docker images · observability · pattern · refactoring to microservices · resilience · sagas · security · service api · service collaboration · service design · service discovery · service granularity · service template · software delivery metrics · success triangle · tacos · team topologies · transaction management · transactional messaging