MERN• 5 MIN READ
Kirito

Kirito

6/10/20265 min read

The Unhandled Promise Rejection That Killed Our Node.js Server

One missing .catch() caused our Node.js server to crash 47 times in one night. Each crash took 30 seconds to recover — 23 minutes of downtime total.

The Unhandled Promise Rejection That Killed Our Node.js Server

3:00 AM. PagerDuty wakes me up.

"API is down."

I check the logs. Nothing unusual. Restart the server. Works fine.

10 minutes later: PagerDuty again.

This happened 47 times in one night.

The Symptoms

The server wasn't crashing with an error. It was just… stopping.

[nodemon] app crashed - waiting for file changes before starting...

No stack trace. No error message. Just silent death.

We were using PM2 in production:

pm2 start app.js --name my-api --max-memory-restart 1G

PM2 would restart the server automatically. But each restart took 30 seconds.

47 restarts × 30 seconds = 23.5 minutes of downtime per night.

The Investigation

After the third night, I was desperate.

I added logging everywhere:

// Global error handlers process.on('uncaughtException', (error) => { console.error('Uncaught Exception:', error); process.exit(1); }); process.on('unhandledRejection', (reason, promise) => { console.error('Unhandled Rejection:', reason); // DON'T exit, just log });

The next crash, I finally saw it:

Unhandled Rejection: TypeError: Cannot read property 'id' of undefined
    at /app/src/controllers/orderController.js:47:32
    at processTicksAndRejections (internal/process/task_queues.js:95:5)

An unhandled promise rejection was crashing Node.js.

But wait — Node.js doesn't crash on unhandled rejections anymore (since v15). Right?

The Real Problem

We were running Node.js 14 (deprecated but "stable").

In Node.js 14, unhandled promise rejections would:

  1. Log a warning
  2. Crash the process (if no handler)

But our global handler should have caught it.

Why didn't it?

The Code

Here was the offending code:

// orderController.js - LINE 47 const processOrder = async (req, res) => { const { orderId } = req.params; // Get order and user in parallel const [order, user] = await Promise.all([ Order.findById(orderId), User.findById(req.user.id) ]); // LINE 47 - CRASH HERE const customerName = user.profile.name; // user is undefined! // ... rest of logic };

When User.findById returned null (user deleted but token still valid), the user variable was undefined.

Then user.profile.name threw an error inside an async function.

The error propagated to the caller, which had no .catch():

// route.js app.post('/api/orders/:orderId/process', processOrder); // NO CATCH

In Express, unhandled rejections in route handlers:

  • Node.js 14: Crash the process
  • Node.js 15+: Log warning but keep running

The Fixes

1. Wrap All Route Handlers

// Async wrapper to catch errors const asyncHandler = (fn) => (req, res, next) => { Promise.resolve(fn(req, res, next)).catch(next); }; // Use it app.post('/api/orders/:orderId/process', asyncHandler(processOrder) );

2. Express Error Handling Middleware

// Error handling middleware app.use((err, req, res, next) => { console.error('Error:', err); // Don't expose internal errors res.status(500).json({ error: 'Internal server error', message: process.env.NODE_ENV === 'development' ? err.message : undefined }); });

3. Fix the Null Check

const processOrder = async (req, res) => { const { orderId } = req.params; const [order, user] = await Promise.all([ Order.findById(orderId), User.findById(req.user.id) ]); // Check for null/undefined if (!order) { return res.status(404).json({ error: 'Order not found' }); } if (!user) { return res.status(404).json({ error: 'User not found' }); } // Now safe const customerName = user.profile.name; // ... rest };

4. Upgrade Node.js

# From 14 to 18 nvm install 18 nvm use 18 # Update dependencies npm update # Test npm test

The Prevention

Global Unhandled Rejection Handler (with monitoring)

// Complete rejection handling const unhandledRejections = new Map(); process.on('unhandledRejection', (reason, promise) => { const error = { message: reason.message || String(reason), stack: reason.stack, timestamp: new Date().toISOString(), promise: String(promise) }; // Store for debugging unhandledRejections.set(Date.now(), error); // Send to Sentry Sentry.captureException(reason, { tags: { type: 'unhandledRejection' }, extra: { promise } }); // Alert if too many if (unhandledRejections.size > 10) { sendAlert({ level: 'critical', message: 'Too many unhandled rejections', count: unhandledRejections.size }); } // In development, log loudly if (process.env.NODE_ENV === 'development') { console.error('💥 UNHANDLED REJECTION 💥'); console.error(reason); } }); // Clean up old entries periodically setInterval(() => { const now = Date.now(); for (const [timestamp] of unhandledRejections) { if (now - timestamp > 24 * 60 * 60 * 1000) { unhandledRejections.delete(timestamp); } } }, 60 * 60 * 1000);

ESLint Rule to Catch Missing Error Handling

// .eslintrc.json { "rules": { "no-floating-promises": "error", "require-await": "error" }, "parserOptions": { "project": "./tsconfig.json" } }

Automatic Testing for Null/Undefined

// Test that catches missing null checks describe('Order Controller', () => { it('should handle missing user gracefully', async () => { // Mock User.findById to return null User.findById.mockResolvedValue(null); const req = { params: { orderId: '123' }, user: { id: 'deleted-user' } }; const res = { status: jest.fn().mockReturnThis(), json: jest.fn() }; await processOrder(req, res); expect(res.status).toHaveBeenCalledWith(404); expect(res.json).toHaveBeenCalledWith({ error: 'User not found' }); }); });

The Monitoring Stack

After the incident, we set up:

1. PM2 Monitoring

# Watch for restarts pm2 monitor pm2 logs --lines 100 # Alert on restart pm2 set pm2-auto-pull:alert-webhook https://hooks.slack.com/...

2. Health Checks

// Every 30 seconds setInterval(async () => { try { await fetch('http://localhost:3000/health'); } catch (error) { // No response - server is down sendAlert('Server not responding'); } }, 30000);

3. Memory Leak Detection

// Monitor memory usage setInterval(() => { const used = process.memoryUsage(); const heapUsedMB = used.heapUsed / 1024 / 1024; const heapTotalMB = used.heapTotal / 1024 / 1024; if (heapUsedMB > 512) { console.warn(`High memory usage: ${heapUsedMB}MB`); if (heapUsedMB > 1024) { sendAlert({ level: 'warning', message: 'Memory leak detected', heapUsed: heapUsedMB }); } } }, 60000);

The Root Cause Analysis

Why did this happen in the first place?

  1. No TypeScript → Could have caught user.profile being accessed on possibly-null value
  2. No async handler wrapper → Express doesn't catch promise rejections by default
  3. Old Node.js version → v14 crashes on unhandled rejections
  4. Insufficient testing → No tests for "user not found" scenario

The Results After Fixes

  • Crash frequency: 47/night → 0
  • Error response time: Crash + 30s restart → 2ms (returns 404)
  • Sentry errors: 200/day → 5/day (mostly handled)
  • Developer confidence: "I hate this server" → "It's stable now"

Commands for Debugging

Find unhandled rejections in code

# Grep for Promise chains without catch grep -r ".then(" --include="*.js" | grep -v ".catch" # Find async functions without try/catch grep -r "async (" --include="*.js" -A 10 | grep -v "try"

Test Node.js version behavior

# Test unhandled rejection behavior node -e "Promise.reject('test')" # Node 14: crashes # Node 16+: logs warning # Run with flags node --unhandled-rejections=throw app.js # Crash node --unhandled-rejections=strict app.js # Log and exit node --unhandled-rejections=none app.js # Ignore (dangerous)

Monitor process uptime

# Check last restart time pm2 list pm2 describe my-api # Check systemd restart count systemctl status my-api | grep "started"

What I Learned

1. Always Wrap Async Express Handlers

// Don't do this app.get('/route', async (req, res) => { ... }) // Do this app.get('/route', asyncHandler(async (req, res) => { ... }))

2. Upgrade Node.js Regularly

LTS releases have critical behavior changes.

3. Test Error Scenarios

Every "findById" can return null. Test it.

4. Monitor Restarts

If your server is restarting frequently, something is wrong.

The Happy Ending

That weekend of 47 crashes taught our team:

  • Never trust async functions to handle their own errors
  • Use TypeScript or JSDoc with strict null checks
  • Test the unhappy path
  • Keep Node.js updated

Now our server has been running for 127 days without a crash.

And I finally disabled PagerDuty's 3 AM alerts for "API is down."