The Kubernetes Rollout That Almost Ruined Black Friday

Black Friday. 6:00 AM. I was on call.

Our e-commerce platform was doing $120,000 per hour. Everything was smooth.

Then I got a Slack message: "Can you update the coupon code service? New promo starts at 7 AM."

"No problem," I thought. "It's just a restart."

The Change

The service was a simple Node.js app running on Kubernetes:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: coupon-service
spec:
  replicas: 5
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
  template:
    spec:
      containers:
      - name: coupon-service
        image: coupon-service:latest
        ports:
        - containerPort: 3000
        # NO readinessProbe
        # NO livenessProbe

I ran:

kubectl rollout restart deployment/coupon-service

The Cascade Failure

Here's what happened next:

T+0 seconds: Kubernetes started terminating old pods.

T+30 seconds: Old pods terminated. New pods started.

T+45 seconds: New pods were running — but not ready.

The Node.js app needed to:

Connect to Redis
Load coupon data from PostgreSQL
Warm up caches

This took 90 seconds total.

But Kubernetes didn't know that.

T+60 seconds: Kubernetes marked pods as "Running" (container started = ready).

Traffic started flowing to pods that weren't ready.

T+75 seconds: The first request hit an unprepared pod.

The pod crashed.

Kubernetes restarted it.

T+90 seconds: Crash loop began.

T+120 seconds: All 5 pods were crash-looping.

Zero healthy replicas.

The Outage

The coupon service was a dependency for:

Cart calculations
Checkout flow
Price displays

When it went down:

Error: Coupon service unavailable

Every cart, every checkout, every page trying to display a price.

The entire site degraded.

47 minutes later, I figured out what happened and fixed it.

Lost revenue: ~$340,000.

Why It Happened

I didn't understand Kubernetes readiness probes.

A readinessProbe tells Kubernetes when a pod is actually ready to receive traffic.

Without it, Kubernetes assumes the pod is ready as soon as the container starts.

# WHAT I SHOULD HAVE HAD
readinessProbe:
  httpGet:
    path: /health/ready
    port: 3000
  initialDelaySeconds: 30
  periodSeconds: 10
  failureThreshold: 3
  successThreshold: 1

My app had a /health/ready endpoint that returned 200 only when:

Database connected
Redis connected
Caches warmed
All dependencies healthy

But I never configured Kubernetes to use it.

The Emergency Fix

While the site was down, I had to:

1. Stop the crash loop

# Scale to zero
kubectl scale deployment/coupon-service --replicas=0

# Let everything settle
sleep 30

# Scale back up slowly
kubectl scale deployment/coupon-service --replicas=1

2. Wait for manual health check

# Monitor pod logs
kubectl logs -f deployment/coupon-service

# Check if it's actually ready
kubectl exec deployment/coupon-service -- curl localhost:3000/health/ready

3. Add readiness probe without downtime

# Edit deployment
kubectl edit deployment/coupon-service

# Add readinessProbe section
# Save and exit - Kubernetes rolls out new pods with probe

The Root Cause Analysis

After the crisis, I investigated why this happened.

Problem 1: No readiness probe → Traffic sent to unprepared pods

Problem 2: No startup probe → Slow-starting apps crash before ready

Problem 3: No podDisruptionBudget → All pods terminated at once

Problem 4: No circuit breakers → Downstream services kept trying

The Correct Configuration

Here's what I implemented afterward:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: coupon-service
spec:
  replicas: 5
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1        # Only 1 extra pod at a time
      maxUnavailable: 0  # Never go below desired replicas
  template:
    spec:
      containers:
      - name: coupon-service
        image: coupon-service:latest
        ports:
        - containerPort: 3000
        
        # STARTUP probe for slow-starting apps
        startupProbe:
          httpGet:
            path: /health/startup
            port: 3000
          initialDelaySeconds: 0
          periodSeconds: 5
          failureThreshold: 30  # Allow 150 seconds total
        
        # READINESS probe for traffic routing
        readinessProbe:
          httpGet:
            path: /health/ready
            port: 3000
          initialDelaySeconds: 0  # Startup probe handles initial delay
          periodSeconds: 5
          failureThreshold: 3
          successThreshold: 1
        
        # LIVENESS probe for crash detection
        livenessProbe:
          httpGet:
            path: /health/live
            port: 3000
          initialDelaySeconds: 60
          periodSeconds: 30
          failureThreshold: 3
        
        resources:
          requests:
            memory: "256Mi"
            cpu: "250m"
          limits:
            memory: "512Mi"
            cpu: "500m"
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: coupon-service-pdb
spec:
  minAvailable: 3
  selector:
    matchLabels:
      app: coupon-service

The Application Changes

I also fixed the app to handle signals properly:

// Graceful shutdown in Node.js
process.on('SIGTERM', async () => {
  console.log('SIGTERM received, starting graceful shutdown');
  
  // Stop accepting new requests
  server.close(() => {
    console.log('HTTP server closed');
  });
  
  // Close database connections
  await db.disconnect();
  await redis.quit();
  
  console.log('Graceful shutdown complete');
  process.exit(0);
});

// Health endpoints
app.get('/health/startup', (req, res) => {
  // Returns 200 as soon as app starts
  res.send('ok');
});

app.get('/health/ready', async (req, res) => {
  const checks = await Promise.all([
    db.ping(),
    redis.ping(),
    cacheWarmed()
  ]);
  
  if (checks.every(c => c === true)) {
    res.send('ready');
  } else {
    res.status(503).send('not ready');
  }
});

What I Learned

1. Readiness Probes Are Not Optional

Every deployment needs proper probes. Every single one.

2. Test Rollouts in Staging With Real Traffic

# Simulate rollout during load test
kubectl rollout restart deployment/coupon-service
# Watch for errors
kubectl get events --watch

3. Use Progressive Rollouts

# Don't restart all at once
kubectl rollout restart deployment/coupon-service   --dry-run=client -o yaml |   kubectl patch -f - --type='json' -p='[{"op": "replace", "path": "/spec/strategy/rollingUpdate/maxUnavailable", "value": 0}]'

4. Implement PodDisruptionBudget

Always set minimum available replicas during voluntary disruptions.

Commands for Safe Rollouts

Check pod readiness

# Wait for rollout to complete
kubectl rollout status deployment/coupon-service --timeout=5m

# Check readiness of each pod
kubectl get pods -l app=coupon-service -o json |   jq '.items[].status.conditions[] | select(.type=="Ready") | .status'

Simulate a rollout

# Dry run
kubectl rollout restart deployment/coupon-service --dry-run=client

# Watch in real-time
kubectl get pods -l app=coupon-service -w

Rollback instantly

# Undo the rollout
kubectl rollout undo deployment/coupon-service

# Check history
kubectl rollout history deployment/coupon-service

Conclusion

That Black Friday cost me weeks of sleepless nights.

The company kept me, but I was written up.

Now I have a pre-rollout checklist:

[ ] Readiness probe configured
[ ] Liveness probe configured
[ ] Startup probe configured (for slow apps)
[ ] PodDisruptionBudget exists
[ ] RollingUpdate strategy has maxUnavailable=0
[ ] Graceful shutdown implemented
[ ] Load tested in staging

Kubernetes gives you powerful tools. But missing one YAML field can cost millions.

Never roll out without probes. Never.