multi-architecture docker images   docker  

Configuring a CircleCI-based pipeline to build multi-architecture Docker images

This is the third article about my adventures trying to use my Apple M1 MacBook for development. In the previous article, I covered how to use the docker build buildx command to create a multi-architecture Docker image. In this article, I describe how a CircleCI CI/CD pipeline can use docker build buildx to build a multi-architecture image and push it to a remote registry.

The other articles in this series are:

Running docker buildx build on CircleCI: first attempt

The microservice-canvas/plantuml has a simple CircleCI pipeline that runs docker build to build the image, tests it locally and then pushes it to Docker Hub. I thought a good way to start was to add a new step to the job that simply runs the build-and-test-multi-arch-locally.sh, which I described in the previous article:

- run:
    name: docker buildx build
    command: |
      docker buildx create --use
      ./build-and-test-multi-arch-circleci.sh
- run: ./build-and-test-docker.sh

This step runs prior to original build script with the goal of creating an image in the local registry container. However, when pipeline ran, it failed with a surprising new error:

error: could not create a builder instance with TLS data loaded from environment. Please use `docker context create <context-name>` to create a context for current environment and then create a builder instance with `docker buildx create <context-name>`

The solution was to create Docker context and create a builder that uses that context.

- run:
    name: docker buildx build
    command: |
      docker context create tls-env
      docker buildx create tls-env --use
      ./build-and-test-multi-arch-circleci.sh
- run: ./build-and-test-docker.sh

This time docker buildx build built the image. But the command then failed to push it to the registry:

=> => pushing layers                                                      0.0s
------
> exporting to image:
------
error: failed to solve: failed to do request: Head "http://host.docker.internal:5002/v2/plantuml/blobs/sha256:69d06718b798aebf8c13b1a53299d1f791f8118dcd0fa9a0a827bf2f595aa9ca": dial tcp: lookup host.docker.internal on 172.28.0.2:53: no such host

The CircleCI remote docker environment doesn’t support the special DNS name host.docker.internal. But what’s even worse is that the CircleCI remote docker environment does not provide network access between the Docker containers and the host machine. Its primarily use case is building Docker images and pushing them to remote registries. Consequently, it’s not clear how to push or pull an image to/from a registry container Furthermore, even if it could be done, the registry is local to a single CircleCI job. It can’t be used for a multi-job pipeline tests the image on multiple architectures.

Pushing the multi-architecture image to a remote registry

The solution is to push the newly created image to a remote registry. But which registry and which repository? I’d like to only use Docker Hub for released images. But for reasons that I’ll describe in a later article, the only solution I could find that supports publishing tested image to Docker Hub is to push the new image to microservicesio/plantuml using a test-build-* tag that indicates it’s work-in-progress!

To implement these changes I wrote a build-and-test-multi-arch-circleci.sh that’s similar to build-and-test-multi-arch-locally.sh except that it uses a remote registry:

TARGET_IMAGE=microservicesio/plantuml:test-build-${CIRCLE_SHA1?}

docker login ...

docker buildx create tls-env --use

docker buildx build --platform linux/amd64,linux/arm64 \
   -t ${TARGET_IMAGE}  --push ...

docker run -i --rm --net=none ${TARGET_IMAGE} ...

When I ran this script, it pushed and then tested the image microservicesio/plantuml:test-build-13559a740bb0eba57bdebe50871b3188ad8ed4a5. Since this image is in a remote repository, I was also able to run it on my M1 MacBook! As expected, it ran without any problems. The docker inspect microservicesio/plantuml:test-build-1355... command showed that image’s architecture was arm64.

In the next article, I’ll describe how to enhance the CircleCI pipeline to test the image on an ARM platform.

Viewing the changes

To see the changes I made to the project, take a look at this Github commit.


multi-architecture docker images   docker  


Copyright © 2023 Chris Richardson • All rights reserved • Supported by Kong.

About Microservices.io

Microservices.io is brought to you by Chris Richardson. Experienced software architect, author of POJOs in Action, the creator of the original CloudFoundry.com, and the author of Microservices patterns.

Chris helps clients around the world adopt the microservice architecture through consulting engagements, and training workshops.

MICROSERVICES WORKSHOPS

Chris teaches comprehensive workshops for architects and developers that will enable your organization use microservices effectively.

Avoid the pitfalls of adopting microservices and learn essential topics, such as service decomposition and design and how to refactor a monolith to microservices.

Learn more

LEARN about microservices

Chris offers numerous other resources for learning the microservice architecture.

Get the book: Microservices Patterns

Read Chris Richardson's book:

Example microservices applications

Want to see an example? Check out Chris Richardson's example applications. See code

Remote consulting session

Got a specific microservice architecture-related question? For example:

  • Wondering whether your organization should adopt microservices?
  • Want to know how to migrate your monolith to microservices?
  • Facing a tricky microservice architecture design problem?

Consider signing up for a two hour, highly focussed, consulting session.

Virtual bootcamp: Distributed data patterns in a microservice architecture

My virtual bootcamp, distributed data patterns in a microservice architecture, is now open for enrollment!

It covers the key distributed data management patterns including Saga, API Composition, and CQRS.

It consists of video lectures, code labs, and a weekly ask-me-anything video conference repeated in multiple timezones.

The regular price is $395/person but use coupon MECNPWNR to sign up for $120 (valid until May 16th, 2023). There are deeper discounts for buying multiple seats.

Learn more

Learn how to create a service template and microservice chassis

Take a look at my Manning LiveProject that teaches you how to develop a service template and microservice chassis.

Signup for the newsletter


BUILD microservices

Ready to start using the microservice architecture?

Consulting services

Engage Chris to create a microservices adoption roadmap and help you define your microservice architecture,


The Eventuate platform

Use the Eventuate.io platform to tackle distributed data management challenges in your microservices architecture.

Eventuate is Chris's latest startup. It makes it easy to use the Saga pattern to manage transactions and the CQRS pattern to implement queries.

ASSESS your architecture

Assess your application's microservice architecture and identify what needs to be improved.

Consulting services

Engage Chris to conduct an architectural assessment.



Join the microservices google group

Topics

Note: tagging is work-in-process

anti-patterns   ·  application api   ·  application architecture   ·  architecting   ·  architecture documentation   ·  assemblage   ·  beer   ·  containers   ·  dark energy and dark matter   ·  deployment   ·  design-time coupling   ·  development   ·  devops   ·  docker   ·  eventuate platform   ·  glossary   ·  hexagonal architecture   ·  implementing commands   ·  implementing queries   ·  inter-service communication   ·  kubernetes   ·  loose coupling   ·  microservice architecture   ·  microservice chassis   ·  microservices adoption   ·  microservicesio updates   ·  multi-architecture docker images   ·  observability   ·  pattern   ·  refactoring to microservices   ·  resilience   ·  sagas   ·  security   ·  service api   ·  service collaboration   ·  service design   ·  service discovery   ·  service granularity   ·  service template   ·  software delivery metrics   ·  success triangle   ·  tacos   ·  team topologies   ·  transaction management   ·  transactional messaging

All content


Posts

24 Jul 2017 » Revised data patterns