Editor’s note: The Docker+Wasm Technical Preview is now available. Find out more about the preview and try it for yourself!

—

You may have seen some hype around WebAssembly, or Wasm, as it’s often called. It’s a relatively new technology that allows you to compile application code written in languages like Rust, C, C++, Javascript, and Golang to byte code, then run it inside a sandboxed environment.

So why all the hype? Well, those sandboxed environments can run in a large variety of locations — including your web browser using a Javascript virtual machine. Not only does this mean the sandbox benefits from billions of dollars of investment in security, speed, and cross compatibility, it also means you can run existing code in your browser with some minor changes. And before you ask, yes, it can run Doom.

But running Doom in the browser is just one use case. Companies like WasmEdge are using Tensorflow to push the boundaries of what can be run with Wasm. Fermyon is building tools for Wasm to be used in microservices, while Vercel, Fastly, Shopify, and Cloudflare use Wasm for running code at the edge. Figma is using Wasm to provide higher performance in the browser for their application, and their new parent company Adobe is bringing their desktop applications to the Web using Wasm.

If all those examples don’t excite you about what’s possible with Wasm, I’m not sure what will!

How do Docker and Wasm fit together?

So what is Docker doing with Wasm? We see Wasm and containers as complementary technologies. The problem you’re solving will make one or the other more applicable, but they’re compatible, and should work well together in your cloud native application.

It really comes down to the use case. For example, Wasm’s quick startup time is great for short lived operations, and its isolation is a good match when you need strict security guarantees enforceable at the code level. But as of now, it doesn’t have multithreading or garbage collection capabilities, so any use case with those requirements isn’t a good fit. It also requires that you rebuild your software from source to work.

Join Docker at Cloud Native Wasm Day

We’ll be at the Cloud Native Wasm Day NA in Detroit on October 24, as a Diamond sponsor, to talk about how we’re providing developers the tooling they need using development experiences they already know and love.

Justin Cormack, our CTO, will be presenting during the keynote. In his presentation, he’ll talk about how the container, Docker, and cloud native communities are embracing Wasm — and give some insights as to where we can go from here. If you’ve never seen him speak, I highly recommend it!

Michael Yuan (WasmEdge) and I will also be giving a talk to show how WASI and container workloads work together in Docker Desktop. You’ll find out when to use Wasm, the current tooling options for Wasm, and how to use Docker and Wasm together. You can check out the GitHub repo for the talk or download the Docker + Wasm preview to try it for yourself!

If you’re attending the Cloud Native Wasm Day don’t miss our keynote and talk!

With Docker focusing on developers, we’ve been doubling down on the parts of Docker that developers love, like Desktop, Hub, and of course Compose. Millions of developers all over the world use Compose to develop their applications and love its simplicity but there was no simple way to get these applications running in the cloud.

Docker is working to make it easier to get code running in the cloud in two ways. First we moved the Compose specification into a community project. This will allow Compose to evolve with the community so that it may better solve more user needs and ensure that it is agnostic of runtime platform. Second, we’ve been working with Amazon and Microsoft on CLI integrations for Amazon ECS and Microsoft ACI that allow you to use docker compose up to deploy Compose applications directly to the cloud.

While implementing these integrations, we wanted to make sure that existing CLI commands were not impacted. We also wanted an architecture that would make it easy to add new backends and provide SDKs in popular languages. We achieved this with the following architecture:

The Node SDK and Compose CLI parts of this diagram are what we have open sourced today. This architecture is not final and we plan to merge the Compose CLI with the existing CLI at a later time.

Depending on the Docker Context that the user selects, the Compose CLI switches which backend is used for the command or API call. This allows us to pass commands which use existing contexts to the existing CLI transparently. The backend interface abstraction allows the implementation of a backend for any container runtime so that users can get the same Docker CLI UX they know and love for it along with the new APIs and SDK.

The Compose CLI can serve a gRPC API to provide similar functionality to that of the CLI commands. We chose to use gRPC as this allows us to generate high quality SDKs in popular languages like Node.js, Python, and Golang. While we currently only provide a Node SDK that supports single container management on ACI, there are plans to add Compose support, extend it to ECS and other backends, and add other language SDKs in the near future. The Node SDK is already used by VS Code to implement its Docker experience on ACI.

This work wouldn’t have been possible without help from our partners at Microsoft and AWS who helped us build the best possible experience for their respective platforms. Our team has enjoyed working with all of you! From Microsoft we’d specifically like to thank Mike Morton, Karol Zadora-Przylecki, Brandon Waterloo, MacKenzie Olson, and Paul Yuknewicz. From AWS we’d like to thank Carmen Puccio, David Killmon, Sravan Rengarajan, Uttara Sridhar, and David Duffey.

These tools are currently in beta so feedback and pull requests are welcome!

• Compose CLI source: https://github.com/docker/compose-cli
• Node SDK source: https://github.com/docker/node-sdk

To get started working with Compose in the Cloud you can download Docker Desktop here, get a free Hub account to deploy your images from here. Once you have you image saved to Docker Hub you will be able to deploy it to either ECS or ACI, to find out more about how to do this:

• Read about how to use the Amazon ECS integration here: https://docs.docker.com/engine/context/ecs-integration/
• Read about how to use the Microsoft ACI integration here: https://docs.docker.com/engine/context/aci-integration/

Adding a linter

We’d like to automate checking for good programming practices as much as possible so let’s add a linter to our setup. First step is to modify the Dockerfile:

We now have a lint-base stage that is an alias for the golangci-lint image which contains the linter that we would like to use. We then have a lint stage that runs the lint, mounting a cache to the correct place.

As for the unit tests, we can add a lint rule to our Makefile for linting. We can also alias the test rule to run the linter and unit tests:

Adding a CI

Now that we’ve containerized our development platform, it’s really easy to add CI for our project. We only need to run our docker build or make commands from the CI script. To demonstrate this, we’ll use GitHub Actions. To set this up, we can use the following .github/workflows/ci.yaml file:

Notice that the commands we run on the CI are identical to those that we use locally and that we don’t need to do any toolchain configuration as everything is already defined in the Dockerfile!

One last optimization

Performing a COPY will create an extra layer in the container image which slows things down and uses extra disk space. This can be avoided by using RUN --mount and bind mounting from the build context, from a stage, or an image. Adopting this pattern, the resulting Dockerfile is as follows:

The default mount type is a read only bind mount from the context that you pass with the docker build command. This means that you can replace the COPY . . with a RUN --mount=target=. wherever you need the files from your context to run a command but do not need them to persist in the final image.

Instead of separating the Go module download, we could remove this and just use a cache mount for /go/pkg/mod.

Conclusion

This series of posts showed how to put in place an optimized containerized Go development environment and then how to use this same environment on the CI. The only dependencies for those who would like to develop on such a project are Docker and make– the latter being optionally replaced by another scripting language.

You can find the source for this example on my GitHub: https://github.com/chris-crone/containerized-go-dev

You can read more about the experimental Dockerfile syntax here: https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/experimental.md

If you’re interested in build at Docker, take a look at the Buildx repository: https://github.com/docker/buildx

Adding dependencies

Go program from part 1 is very simple and doesn’t have any dependencies Go dependencies. Let’s add a simple dependency – the commonly used github.com/pkg/errors package:

Our example program is now a simple echo program that writes out the arguments that the user inputs or “no message to echo” and a stack trace if nothing is specified.

We will use Go modules to handle this dependency. Running the following commands will create the go.mod and go.sum files:

$ go mod init
$ go mod tidy

Now when we run the build, we will see that each time we build, the dependencies are downloaded

$ make
[+] Building 8.2s (7/9)
 => [internal] load build definition from Dockerfile
...
0.0s
 => [build 3/4] COPY . . 
0.1s
 => [build 4/4] RUN GOOS=darwin GOARCH=amd64 go build -o /out/example . 
7.9s
 => => # go: downloading github.com/pkg/errors v0.9.1 

This is clearly inefficient and slows things down. We can fix this by downloading our dependencies as a separate step in our Dockerfile:

Notice that we’ve added the go.* files and download the modules before adding the rest of the source. This allows Docker to cache the modules as it will only rerun these steps if the go.* files change.

Caching

Separating the downloading of our dependencies from our build is a great improvement but each time we run the build, we are starting the compile from scratch. For small projects this might not be a problem but as your project gets bigger you will want to leverage Go’s compiler cache.

To do this, you will need to use BuildKit’s Dockerfile frontend (https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/experimental.md). Our updated Dockerfile is as follows:

Notice the # syntax at the top of the Dockerfile that selects the experimental Dockerfile frontend and the –mount option attached to the run command. This mount option means that each time the go build command is run, the container will have the cache mounted to Go’s compiler cache folder.

Benchmarking this change for the example binary on a 2017 MacBook Pro 13”, I see that a small code change takes 11 seconds to build without the cache and less than 2 seconds with it. This is a huge improvement!

Adding unit tests

All projects need tests! We’ll add a simple test for our echo function in a main_test.go file:

This test ensures that we get an error if the echo function is passed an empty list of arguments.

We will now want another build target for our Dockerfile so that we can run the tests and build the binary separately. This will require a refactor into a base stage and then unit-test and build stages:

Note that Go test uses the same cache as the build so we mount the cache for this stage too. This allows Go to only run tests if there have been code changes which makes the tests run quicker.

We can also update our Makefile to add a test target:

What’s next?

In this post we have seen how to add Go dependencies efficiently, caching to make the build faster and unit tests to our containerized Go development environment. In the next and final post of the series, we are going to complete our journey and learn how to add a linter, set up a GitHub Actions CI, and some extra build optimizations.

You can find the finalized source this example on my GitHub: https://github.com/chris-crone/containerized-go-dev

You can read more about the experimental Dockerfile syntax here: https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/experimental.md

If you’re interested in build at Docker, take a look at the Buildx repository: https://github.com/docker/buildx

Read the whole blog post series here.

This problem continues as you spend more time in the team. You’ll find issues because the version of the tool you’re using is different to that used by someone on your team, or, worse, the CI. I’ve been on more than one team where “works on my machine” has been exclaimed or written in all caps on Slack and I’ve spent a lot of time debugging things on the CI which is incredibly painful.

Many people use Docker as a way to run application dependencies, like databases, while they’re developing locally and for containerizing their production applications. Docker is also a great tool for defining your development environment in code to ensure that your team members and the CI are all using the same set of tools.

We do a lot of Go development at Docker. The Go toolchain is great– providing fast compile times, built in dependency management, easy cross compiling, and strong opinionation on things like code formatting. Even with this toolchain we often run into issues like mismatched versions of Go, missing dependencies, and slightly different configurations. A good example of this is that we use gRPC for many projects and so require a specific version of protoc that works with our code base.

This is the first of a series of blog posts that will show you how to use Docker for Go development. It will cover building, testing, CI, and optimization to make your builds quicker.

Start simple

Let’s start with a simple Go program:

You can easily build this into a binary using the following command:
$ go build -o bin/example .

The same can be achieved using the following Dockerfile:

This Dockerfile is broken into two stages identified with the AS keyword. The first stage, build, starts from the Go Alpine image. Alpine uses the musl C library and is a minimalist alternative to the regular Debian based Golang image. Note that we can define which version of Go we want to use. It then sets the working directory in the container, copies the source from the host into the container, and runs the go build command from before. The second stage, bin, uses a scratch (i.e.: empty) base image. It then simply copies the resulting binary from the first stage to its filesystem. Note that if your binary needs other resources, like CA certificates, then these would also need to be included in the final image.

As we are leveraging BuildKit in this blog post, you will need to make sure that you enable it by using Docker 19.03 or later and setting DOCKER_BUILDKIT=1 in your environment. On Linux, macOS, or using WSL 2 you can do this using the following command:
$ export DOCKER_BUILDKIT=1

On Windows for PowerShell you can use:
$env:DOCKER_BUILDKIT=1

Or for command prompt:
set DOCKER_BUILDKIT=1

To run the build, we will use the docker build command with the output option to say that we want the result to be written to the host’s filesystem:
$ docker build --target bin --output bin/ .

You will then see that we have the example binary inside our bin directory:

$ ls bin
example

Simple cross compiling

As Docker supports building images for different platforms, the build command has a platform flag. This allows setting the target OS (i.e.: Linux or Windows) and architecture (i.e.: amd64, arm64, etc.). To build natively for other platforms, your builder will need to be set up for these platforms. As Golang’s toolchain includes cross compilation, we do not need to do any builder setup but can still leverage the platform flag.

We can cross compile the binary for the host operating system by adding arguments to our Dockerfile and filling them from the platform flag of the docker build command. The updated Dockerfile is as follows:

Notice that we now have the BUILDPLATFORM variable set as the platform for our base image. This will pin the image to the platform that the builder is running on. In the compilation step, we consume the TARGETOS and TARGETARCH variables, both filled by the build platform flag, to tell Go which platform to build for. To simplify things, and because this is a simple application, we statically compile the binary by setting CGO_ENABLED=0. This means that the resulting binary will not be linked to any C libraries. If your application uses any system libraries (like the system’s cryptography library) then you will not be able to statically compile the binary like this.

To build for your host operating system, you can specify the local platform:
$ docker build --target bin --output bin/ --platform local .

As the docker build command is getting quite long, we can put it into a Makefile (or a scripting language of your choice):

This will allow you to run your build as follows:

$ make bin/example
$ make

We can go a step further and add a cross compiling targets to the Dockerfile:

Above we have two different stages for Unix-like OSes  (bin-unix) and for Windows (bin-windows). We then add aliases for Linux (bin-linux) and macOS (bin-darwin). This allows us to make a dynamic target (bin) that depends on the TARGETOS variable and is automatically set by the docker build platform flag.

This allows us to build for a specific platform:

$ docker build --target bin --platform windows/amd64 .
$ file bin/
bin/example.exe: PE32+ executable (console) x86-64 (stripped to external
PDB), for MS Windows

Our updated Makefile has a PLATFORM variable that you can set:

This means that you can build for a specific platform by setting PLATFORM:
$ make PLATFORM=windows/amd64

You can find the list of valid operating system and architecture combinations here: https://golang.org/doc/install/source#environment.

Shrinking our build context

By default, the docker build command will take everything in the path passed to it and send it to the builder. In our case, that includes the contents of the bin/ directory which we don’t use in our build. We can tell the builder not to do this, using a .dockerignore file:

Since the bin/ directory contains several megabytes of data, adding the .dockerignore file reduces the time it takes to build by a little bit.

Similarly, if you are using Git for code management but do not use git commands as part of your build process, you can exclude the .git/ directory too.

What’s next?

This post showed how to start a containerized development environment for local Go development, build an example CLI tool for different platforms and how to start speeding up builds by shrinking the build context. In the next post of this series, we will add dependencies to make our example project more realistic, look at caching to make builds faster, and add unit tests.

You can find the finalized source this example on my GitHub: https://github.com/chris-crone/containerized-go-dev

If you’re interested in build at Docker, take a look at the Buildx repository: https://github.com/docker/buildx

Read the whole blog post series here.

As the “Docker project” grew from being a proof of concept shown off at various meetups and at PyCon in 2013 to a real community project, it needed a website where people could learn about it and download it. This is why the “dockerproject.org” and “dockerproject.com” domains were registered.

With the move from dotCloud to Docker, Inc. and the shift of focus onto the Docker product, it made sense to move everything to the “docker.com” domain. This is where you now find the company website, documentation, and of course the APT and YUM repositories at download.docker.com have been there since 2017.

On the 31st of March 2020, we will be shutting down the legacy APT and YUM repositories hosted at dockerproject.org and dockerproject.com. These repositories haven’t been updated with the latest releases of Docker and so the packages hosted there contain security vulnerabilities. Removing these repositories will make sure that people download the latest version of Docker ensuring their security and providing the best experience possible

What do I need to do?

If you are currently using the APT or YUM repositories from dockerproject.org or dockerproject.com, please update to use the repositories at download.docker.com.

You can find instructions for CentOS, Debian, Fedora and Ubuntu in the documentation.