Configuring CircleCI to build, test and publish multi-architecture images for Eventuate Common

multi-architecture docker images   docker  

In the previous post, I described the changes I made to the Eventuate Common project to locally build and test the Java libraries with the multi-architecture MySQL8 and Zookeeper images. In this post, you will learn how I configured the CircleCI pipeline to build, test and publish these images. I describe a useful pattern for structuring a pipeline that builds and tests a multi-architecture Docker image:

  1. Build and test a multi-architecture image locally (using a registry container) on both Intel and Arm.
  2. Build a multi-architecture image and push it to Docker Hub with a test-build-* tag
  3. Test the multi-architecture test-build-* image on both Intel and Arm.
  4. Create a manifest in Docker Hub with the desired tag

The other articles in this series are:

Add CircleCI jobs that build and test locally on Intel and ARM

The first change was to replace the original build-and-test-mysql8 with a pair of jobs - build-and-test-mysql8-intel to build-and-test-mysql8-arm - that locally build and test the multi-architecture Docker image on Intel and Arm. The two jobs are almost identical. The only difference is that the build-and-test-mysql8-arm job specifies the arm.medium resource class:

- build-and-test/build-and-test:
    name: build-and-test-mysql8-intel
    script: |
      docker context create tls-env
      docker buildx create tls-env --use
      docker run --privileged --rm tonistiigi/binfmt --install arm64,arm
      export DOCKER_HOST_NAME=$(hostname)
      ./build-and-test-all-mysql8-multi-arch-locally.sh
    skip_multi_arch_env_vars: "true"
- build-and-test/build-and-test:
    name: build-and-test-mysql8-arm
    resource_class: arm.medium
    script: |
      docker context create tls-env
      docker buildx create tls-env --use
      docker run --privileged --rm tonistiigi/binfmt --install amd64
      export DOCKER_HOST_NAME=$(hostname)
      ./build-and-test-all-mysql8-multi-arch-locally.sh
    skip_multi_arch_env_vars: "true"

Both jobs configure the usual docker buildx builder. What’s unusual is that they both run docker run --privileged --rm tonistiigi/binfmt --install to install QEMU emulators. This command is required because without it the docker buildx command fails.

error: failed to solve: rpc error: code = Unknown desc = process "/dev/.buildkit_qemu_emulator /bin/sh -c mkdir -p /usr/local/zookeeper-conf /usr/local/zookeeper-data &&   yum install -y wget tar gzip &&   (wget -q -O - https://archive.apache.org/dist/zookeeper/zookeeper-3.5.6/apache-zookeeper-3.5.6-bin.tar.gz | tar -xzf - -C /usr/local)" did not complete successfully: exit code: 2

Building a multi-platform image executes the commands in the Dockerfile for each architeture. It uses the QEMU emulator for the non-native architectures. And, as I described in my original article, the emulation is not perfect. The Zookeeper Dockerfile needed updated emulators.

Adding a CircleCI job that builds and pushes the test-build images

The second change was a new CircleCI job - build-multi-arch-images - that builds the multi-architecture MySQL8 and Zookeeper images and pushes them to Docker Hub with a test-build-${CIRCLE_SHA1?} tag.

jobs:
  build-multi-arch-images:
    docker:
      - image: cimg/base:stable
    working_directory: ~/eventuate-common
    steps:
      - checkout
      - setup_remote_docker:
          version: 20.10.11
      - run:
          name: docker buildx build
          command: |
            . set-multi-arch-image-env-vars.sh
            docker context create tls-env
            docker buildx create tls-env --use
            docker run --privileged --rm tonistiigi/binfmt --install arm64,arm

            ./build-multi-arch-images.sh

This job runs after the completion of the local build-and-test-mysql8-* jobs. It’s similar to the job I described in a previous article. The only difference is that it runs docker run --privileged --rm tonistiigi/binfmt --install before running docker buildx build.

Defining CircleCI jobs that test the multi-architecture images

The next change was to define a pair of jobs that test the multi-architecture images that the previous job pushed to Docker Hub. Here’s the configuration of the pipeline:

- build-and-test/build-and-test:
    name: test-mysql8-zookeeper-multi-arch-intel
    script: ./build-and-test-all-mysql8-multi-arch.sh
    requires:
      - build-multi-arch-images
- build-and-test/build-and-test:
    name: test-mysql8-zookeeper-multi-arch-arm
    script: ./build-and-test-all-mysql8-multi-arch.sh
    resource_class: arm.medium
    requires:
      - build-multi-arch-images

These jobs are identical except that one runs on Intel and other runs on Arm.

Configuring CircleCI to publish the multi-architecture Docker images with the desired tag

The final change adds a job that uses docker buildx imagetools to create a manifest in Docker Hub with the desired label. Here’s the configuration of the pipeline:

- deploy-multi-arch:
    context:
      - publish
    requires:
      - test-mysql8-zookeeper-multi-arch-intel
      - test-mysql8-zookeeper-multi-arch-arm

Viewing the changes

To see the changes I made to the project, take a look at this Github commit.

Next steps

The MySQL8 and Zookeeper Docker images aren’t the only Docker images used by the Eventuate Tram Customers and Orders example application. In later articles, I’ll describe how I changed other projects to also build multi-architecture images.


multi-architecture docker images   docker  


Copyright © 2024 Chris Richardson • All rights reserved • Supported by Kong.

About Microservices.io

Microservices.io is brought to you by Chris Richardson. Experienced software architect, author of POJOs in Action, the creator of the original CloudFoundry.com, and the author of Microservices patterns.

Upcoming public workshops: Microservices and architecting for fast flow

In-person: Berlin and Milan

DevOps and Team topologies are vital for delivering the fast flow of changes that modern businesses need.

But they are insufficient. You also need an application architecture that supports fast, sustainable flow.

Learn more and register for one of my upcoming public workshops in November.

NEED HELP?

I help organizations improve agility and competitiveness through better software architecture.

Learn more about my consulting engagements, and training workshops.

LEARN about microservices

Chris offers numerous other resources for learning the microservice architecture.

Get the book: Microservices Patterns

Read Chris Richardson's book:

Example microservices applications

Want to see an example? Check out Chris Richardson's example applications. See code

Virtual bootcamp: Distributed data patterns in a microservice architecture

My virtual bootcamp, distributed data patterns in a microservice architecture, is now open for enrollment!

It covers the key distributed data management patterns including Saga, API Composition, and CQRS.

It consists of video lectures, code labs, and a weekly ask-me-anything video conference repeated in multiple timezones.

The regular price is $395/person but use coupon CCMHVSFB to sign up for $95 (valid until November 8th, 2024). There are deeper discounts for buying multiple seats.

Learn more

Learn how to create a service template and microservice chassis

Take a look at my Manning LiveProject that teaches you how to develop a service template and microservice chassis.

Signup for the newsletter


BUILD microservices

Ready to start using the microservice architecture?

Consulting services

Engage Chris to create a microservices adoption roadmap and help you define your microservice architecture,


The Eventuate platform

Use the Eventuate.io platform to tackle distributed data management challenges in your microservices architecture.

Eventuate is Chris's latest startup. It makes it easy to use the Saga pattern to manage transactions and the CQRS pattern to implement queries.


Join the microservices google group