DevOps• 5 MIN READ
Kirito

Kirito

6/10/20265 min read

The 2.3GB Docker Image That Killed Our Deploys

A Node.js Docker image grew to 2.3GB, causing 25-minute deploys and $500/day in extra costs — until we discovered multi-stage builds.

The 2.3GB Docker Image That Killed Our Deploys

"Deploying..."

That word started giving me anxiety.

Every deploy took 25 minutes. Our CI/CD pipeline was clogged. Developers waited hours for their changes to reach production.

The culprit? A Docker image that weighed 2.3 gigabytes.

How We Got Here

Our Dockerfile was "simple":

# THE HORROR DOCKERFILE FROM node:18 WORKDIR /app COPY package*.json ./ RUN npm ci COPY . . RUN npm run build EXPOSE 3000 CMD ["npm", "start"]

Seems fine, right?

Here's what was actually happening:

Layer 1: Node base image

node:18-slim: 250MB

Layer 2: npm ci

node_modules: 800MB

Layer 3: Copy source code

Source + build artifacts: 50MB

Layer 4: Build output

.next/ (Next.js build): 200MB
dist/ (TypeScript output): 100MB

Total: ~1.4GB

Wait, I said 2.3GB earlier. Where did the extra 900MB come from?

Cached layers.

Each time we rebuilt, Docker added new layers instead of replacing them.

After 10 builds: 2.3GB.

The Impact

Storage Costs

We were running 20 microservices.

Each service had 5 versions stored in ECR.

2.3GB × 20 services × 5 versions = 230GB
ECR cost: 230GB × $0.10/GB/month = $23/month

Not terrible, actually.

The Real Cost: Time

25 minutes per deploy × 50 deploys/day = 20.8 hours of waiting daily

Developer time cost: 20.8 hours × $100/hour = $2,080/day

$500,000/year in wasted productivity.

The Analysis

I used docker history to see what was taking space:

docker history myapp:latest IMAGE CREATED SIZE a1b2c3d4 2 hours ago 523MB # Build output e5f6g7h8 2 hours ago 812MB # npm ci (dev dependencies) i9j0k1l2 3 hours ago 250MB # Node base m3n4o5p6 3 hours ago 0B # WORKDIR

The problem: Development dependencies and build tools in production.

  • webpack, typescript, @types/*: 400MB
  • eslint, prettier, jest: 150MB
  • Source maps: 200MB
  • Duplicate dependencies: 150MB

The Solution: Multi-Stage Builds

# Stage 1: Build FROM node:18-alpine AS builder WORKDIR /app # Install dependencies (including dev) COPY package*.json ./ RUN npm ci # Copy source and build COPY . . RUN npm run build # Stage 2: Production FROM node:18-alpine AS runner WORKDIR /app # Copy only production dependencies COPY package*.json ./ RUN npm ci --only=production # Copy built artifacts from builder COPY --from=builder /app/dist ./dist COPY --from=builder /app/.next ./.next # For Next.js # Copy necessary configs COPY --from=builder /app/package.json ./ EXPOSE 3000 CMD ["node", "dist/index.js"]

The Results

New Image Size: 180MB

From 2.3GB to 180MB = 92% reduction

Deploy Time: 25 minutes → 90 seconds

Pull time: 2.3GB @ 100Mbps = 3 minutes
Pull time: 180MB @ 100Mbps = 14 seconds

Storage Cost: $23/month → $1.80/month

Even Better: Alpine + Distroless

We pushed further:

# Stage 1: Build (using slim instead of alpine for better compatibility) FROM node:18-slim AS builder RUN apt-get update && apt-get install -y python3 make g++ WORKDIR /app COPY package*.json ./ RUN npm ci COPY . . RUN npm run build # Stage 2: Production distroless FROM gcr.io/distroless/nodejs18-debian11 WORKDIR /app # Copy production dependencies COPY --from=builder /app/node_modules ./node_modules COPY --from=builder /app/dist ./dist COPY --from=builder /app/package.json ./ USER nonroot EXPOSE 3000 CMD ["dist/index.js"]

Final size: 95MB

Optimization Techniques

1. Layer Caching

# Bad: Cache invalidated when ANY file changes COPY . . RUN npm ci # Good: Package.json changes rarely COPY package*.json ./ RUN npm ci # Cached unless package.json changes COPY . .

2. .dockerignore

.git node_modules npm-debug.log .env .DS_Store coverage .nyc_output *.log

3. Combine RUN commands

# Bad: Multiple layers RUN apt-get update RUN apt-get install -y curl RUN apt-get clean # Good: Single layer RUN apt-get update && apt-get install -y curl && apt-get clean && rm -rf /var/lib/apt/lists/*

4. Use --link for faster builds (BuildKit)

DOCKER_BUILDKIT=1 docker build --link --tag myapp .

The Monitoring Setup

We started tracking image sizes:

#!/bin/bash # measure-image-size.sh IMAGE_SIZE=$(docker inspect myapp:latest --format='{{.Size}}' | numfmt --to=iec) echo "Image size: $IMAGE_SIZE" # Alert if > 500MB if [ $(docker inspect myapp:latest --format='{{.Size}}') -gt 524288000 ]; then echo "WARNING: Image size exceeded 500MB" # Send to Slack curl -X POST -H 'Content-type: application/json' --data "{"text":"🚨 Docker image size is {IMAGE_SIZE}"}" $SLACK_WEBHOOK fi

The Ultimate Dockerfile Template

Here's what we use for all Node.js services now:

# syntax=docker/dockerfile:1.4 FROM node:18-alpine AS builder RUN apk add --no-cache python3 make g++ WORKDIR /build # Install dependencies COPY package.json package-lock.json ./ RUN --mount=type=cache,target=/root/.npm npm ci --cache=/root/.npm # Build COPY . . RUN npm run build RUN npm prune --production # Production FROM node:18-alpine RUN addgroup -g 1001 -S nodejs && adduser -S nodejs -u 1001 WORKDIR /app COPY --from=builder --chown=nodejs:nodejs /build/package.json ./ COPY --from=builder --chown=nodejs:nodejs /build/node_modules ./node_modules COPY --from=builder --chown=nodejs:nodejs /build/dist ./dist USER nodejs EXPOSE 3000 HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 CMD node -e "require('http').get('http://localhost:3000/health', (r) => {process.exit(r.statusCode === 200 ? 0 : 1)})" CMD ["node", "dist/index.js"]

Commands for Docker Image Optimization

Analyze image size

# See layer sizes docker history myapp:latest # Detailed analysis with dive dive myapp:latest # Export and analyze docker save myapp:latest | tar tvz | sort -k3 -n

Clean up unused images

# Remove dangling images docker image prune # Remove all unused images docker image prune -a # Show space usage docker system df

Build with optimizations

# Enable BuildKit export DOCKER_BUILDKIT=1 # Build with compression docker build --compress --tag myapp . # Squash layers (experimental) docker build --squash --tag myapp .

What I Learned

1. Base Images Matter

  • node:18: 1.1GB
  • node:18-slim: 250MB
  • node:18-alpine: 170MB
  • distroless/nodejs: 80MB

2. Multi-Stage Builds Are Non-Negotiable

Always separate build environment from runtime.

3. Production Doesn't Need Dev Tools

npm ci --only=production is your friend.

4. Monitor Image Growth

Set alerts when images exceed thresholds.

The Happy Ending

After implementing multi-stage builds across all services:

  • Average image size: 2.1GB → 120MB (94% reduction)
  • Average deploy time: 25 min → 2 min
  • CI/CD pipeline: From 3 hours to 15 minutes
  • Developer satisfaction: "Deploys don't make me want to quit anymore"

That 2.3GB Docker image cost us 6 months of developer productivity.

Now we have a "No image over 500MB" rule enforced in CI.

And I finally sleep through deployments.