Microservice Architecture

Supported by Kong
multi-architecture docker images   docker  

Configuring CircleCI to publish a multi-architecture Docker image

This is the fifth article about my adventures trying to use my Apple M1 MacBook for development. In the previous article, I described how to configure a CircleCI-based CI/CD pipeline to test a Docker image on both Intel and ARM platforms. Currently, however, the pipeline doesn’t do anything with the image once the Arm-platform tests pass. It leaves the image in Docker Hub with the tag test-build-${CIRCLE_SHA1?}. In this article, I describe how to configure the pipeline to add a ‘proper’ tag (e.g. BUILD-*, x.y.z.RELEASE, etc) to the image without rebuilding it.

The other articles in this series are:

Defining a deploy job

The pipeline’s build job is still running the original deploy-artifacts.sh script which pushes the single architecture image to Docker Hub before testing the multi-architecture image on Arm. This no longer makes sense and so the first change is to define deploy job, which only runs deploy-artifacts.sh if the test-arm64 job succeeds:

...
deploy:
  docker:
    - image: cimg/base:stable
  working_directory: ~/plantuml
  steps:
    - checkout
    - setup_remote_docker:
        version: 20.10.11
    - run:
        command: ./deploy-artifacts.sh
workflows:
version: 2.1
build-test-and-deploy:
  jobs:
    - build
    - test-arm64:
        requires:
          - build
    - deploy:
        requires:
          - test-arm64

The deploy job executes once the test-arm64 jobs succeeds. Let’s now look at how to publish the multi-architecture with the correct tag.

Publishing the image with the desired tag

The original deploy-artifacts.sh script simply pushed the locally built image to Docker Hub that had a tag corresponding to the Git branch. For example, a master branch build would push a BUILD-${CIRCLE_BUILD_NUM?} tag, a x.y.z.RELEASE branch build would push a x.y.z.RELEASE tag. What’s different about building a multi-architecture image is that it’s already been pushed to Docker Hub with a test-build-${CIRCLE_SHA1?} tag.

I naively thought that I could assign the correct tag by simply executing the following sequence of commands:

  1. docker pull
  2. docker tag to add a desired tag
  3. docker push

These commands successfully publish an appropriately tagged image. However, that image is an Intel-only image! Yet another reminder that a Docker daemon only supports a single architecture. Pulling a multi-architecture image only downloads an image for that architecture.

One solution would be to rebuild the multi-architecture image with the desired tag. But, one drawback of rebuilding the image is that it publishes an image that hasn’t been tested

After a lot of googling, I discovered that I could use the docker manifest command to ‘add’ the desired tag to the image. As I described in Part 2 - Building multi-architecture Docker images for Intel and ARM, a manifest for a multi-architecture image is a JSON object that points to a collection of architecture-specific images. For example:

% docker manifest inspect microservicesio/plantuml:test-build-11c53...
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1994,
         "digest": "sha256:...",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1994,
         "digest": "sha256:28899af3cb9a1756149adef730ba0596d4ea334727d5a2f47d26746c066f20b3",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      }
   ]
}

This manifest, for example, specifies that the Arm-specific image is microservicesio/plantuml@sha256:28899af3cb9a1756149adef730ba0596d4ea334727d5a2f47d26746c066f20b3.

Adding the desired tag to an image is simply a matter of using docker manifest create to create a new manifest that references the previously created images and docker manifest push to push the manifest to Docker Hub.

Using docker buildx imagetools create to tag the image

I then stumbled across a simpler way to create and push a manifest using the docker buildx imagetools create command. The docker buildx imagetools provides a set of commands for working on images in a registry. For example, this command creates the specified multi-architecture image in Docker Hub from the previously pushed images.

docker buildx imagetools create -t microservicesio/plantuml:BUILD-999 \
  microservicesio/plantuml@sha256:28899af3cb9a1756149adef730ba0596d4ea334727d5a2f47d26746c066f20b3 ...

Since it’s manipulating JSON metadata, the command is extremely fast.

The deploy-artifacts.sh script invokes docker buildx imagetools create with the source images obtained by using jq. The jq commands extract the digests from the JSON manifest and turns them into image references:

SOURCES=$(docker manifest inspect docker.io/microservicesio/plantuml:${SRC_TAG} | \
   jq -r '.manifests[].digest  | sub("^"; "docker.io/microservicesio/plantuml@")')

docker buildx imagetools create -t ${TARGET_IMAGE} $SOURCES

Since none of these commands use the Docker Daemon, the deploy job does not need setup_remote_docker.

After making these changes, the master branch build published microservicesio/plantuml:BUILD-131 image. I then created the 0.3.0.RELEASE branch. Finally, I’m able to run PlantUML on my M1 MacBook!

Viewing the changes

To see the changes I made to the project, take a look at this Github commit.

Next steps

Configuring the CircleCI pipeline to publish a multi-architecture Docker image was great learning experience. It also enabled me to use PlantUML on my M1 Macbook! The next step is to apply the lessons I learned here to the Eventuate projects.

multi-architecture docker images   docker  
Copyright © 2023 Chris Richardson • All rights reserved • Supported by Kong.

About Microservices.io

Microservices.io is brought to you by Chris Richardson. Experienced software architect, author of POJOs in Action, the creator of the original CloudFoundry.com, and the author of Microservices patterns.

Chris helps clients around the world adopt the microservice architecture through consulting engagements, and training workshops.

Learn how to create a service template and microservice chassis

Take a look at my Manning LiveProject that teaches you how to develop a service template and microservice chassis.

New virtual bootcamp: Distributed data patterns in a microservice architecture

My virtual bootcamp, distributed data patterns in a microservice architecture, is now open for enrollment!

It covers the key distributed data management patterns including Saga, API Composition, and CQRS.

It consists of video lectures, code labs, and a weekly ask-me-anything video conference repeated in multiple timezones.

The regular price is $395/person but use coupon JUNVCEJE to sign up for $195 (valid until February 1st, 2023). There are deeper discounts for buying multiple seats.

Learn more

Signup for the newsletter


LEARN about microservices

Chris offers numerous resources for learning the microservice architecture.

Training classes

Chris teaches comprehensive workshops, training classes and bootcamps for executives, architects and developers to help your organization use microservices effectively.

Avoid the pitfalls of adopting microservices and learn essential topics, such as service decomposition and design and how to refactor a monolith to microservices.

Delivered in-person and remotely.

Get the book: Microservices Patterns

Read Chris Richardson's book:

Example microservices applications

Want to see an example? Check out Chris Richardson's example applications. See code

BUILD microservices

Ready to start using the microservice architecture?

Consulting services

Engage Chris to create a microservices adoption roadmap and help you define your microservice architecture,

The Eventuate platform

Use the Eventuate.io platform to tackle distributed data management challenges in your microservices architecture.

Eventuate is Chris's latest startup. It makes it easy to use the Saga pattern to manage transactions and the CQRS pattern to implement queries.

ASSESS your architecture

Assess your application's microservice architecture and identify what needs to be improved.

Consulting services

Engage Chris to conduct an architectural assessment.

Join the microservices google group

Topics

Note: tagging is work-in-process

anti-patterns   ·  application api   ·  application architecture   ·  architecting   ·  architecture documentation   ·  dark energy and dark matter   ·  deployment   ·  development   ·  devops   ·  docker   ·  implementing commands   ·  implementing queries   ·  inter-service communication   ·  loose coupling   ·  microservice architecture   ·  microservice chassis   ·  microservices adoption   ·  microservicesio updates   ·  multi-architecture docker images   ·  observability   ·  pattern   ·  refactoring to microservices   ·  resilience   ·  sagas   ·  security   ·  service api   ·  service collaboration   ·  service design   ·  service discovery   ·  service granularity   ·  service template   ·  software delivery metrics   ·  success triangle   ·  team topologies   ·  transaction management   ·  transactional messaging

All content

Posts

24 Jul 2017 » Revised data patterns