An Explicit Docker Workflow

14 Jul, 2024

Initial Observations and Problem Formulation

I've been reflecting on the friction points in my backend development cycle, specifically when integrating Docker for service-layer testing. The core issue has consistently been a drift between the code on my local machine and the code executing within the container. This divergence leads to two primary problems: testing against stale code and the gradual, often unnoticed, exhaustion of disk space due to dangling images.

An image is effectively a static snapshot. Without a deliberate and consistent process for rebuilding this snapshot, any tests are run against a previous state of the application. The Docker caching mechanism, while useful for performance, can sometimes exacerbate this by obscuring whether a code change has actually been incorporated into a new image layer. This has led to time spent debugging issues that were already fixed locally.

To address this, I've settled on a four-step sequence that has introduced a necessary degree of predictability into the process.

Step 1: Ensuring a Clean State (down)

The first action is always to bring down the existing environment.

docker compose down

Previously, I would often attempt to build and run over the top of existing containers. This frequently resulted in address already in use errors or, more insidiously, situations where tests would interact with orphaned containers from a previous run. Explicitly tearing down the environment ensures a clean slate and eliminates a significant source of test flakiness.

Step 2: Rebuilding the Image (build)

With a clean state, the next step is to rebuild the service image to incorporate the latest code changes.

docker compose build

This command creates a new image containing the current state of the codebase. On occasions where I suspected the Docker cache was persisting an unwanted state—particularly after modifying dependency configurations or build scripts—forcing a rebuild without the cache proved to be a necessary diagnostic tool.

# Utilized when cache behavior is suspect.
docker compose build --no-cache

Step 3: Detached Execution (up --detach)

Once the image is rebuilt, the services are launched in detached mode.

docker compose up --detach

Running containers in the foreground is useful for direct observation of logs, but it occupies the terminal, preventing the execution of test suites or other commands. The --detach flag runs the containers in the background, freeing the terminal and enabling the automation of subsequent steps in a script.

Step 4: Routine System Hygiene (prune)

This final step was adopted after experiencing significant performance degradation and, eventually, Docker daemon instability. Each build operation can leave the previous image untagged and unused—a "dangling" image. Over weeks of development, these accumulate.

I have found it necessary to periodically run a pruning command to reclaim disk space.

docker image prune --all --force

This removes all images not currently associated with a container. Integrating this into my routine has prevented the critical disk space shortages that can halt work and affect system stability.

Consolidated Workflow Reference

This sequence has become my standard procedure for local development and testing. It imposes a discipline that, while adding a few explicit commands, has reduced time lost to environment-related debugging.