The Push Notification That Woke Up 50,000 Users at 3 AM

3:00 AM. My phone buzzes.

Then again.

Then 50,000 times.

I had just deployed a "good morning" notification feature to our Expo app.

The only problem? My cron job was set to UTC, not local time.

The Feature

We built a wellness app with daily reminders:

Morning motivation quotes
Water drinking reminders
Step goal achievements

The code was simple:

// Notification service
const sendMorningNotification = async () => {
  const users = await getUserTimezones();
  
  for (const user of users) {
    const localHour = getLocalHour(user.timezone);
    
    // Send at 8 AM local time
    if (localHour === 8) {
      await sendPushNotification(user.expoPushToken, {
        title: "Good morning! 🌅",
        body: "Ready to make today great?",
        data: { screen: "Dashboard" },
        sound: "default"
      });
    }
  }
};

And the cron job:

# WRONG - Runs at 8 AM UTC (3 AM EST)
0 8 * * * node send-morning-notifications.js

The Disaster

At 8:00 UTC (3:00 AM EST for New York users):

[BATCH 1] Sending to 12,347 users in EST
[BATCH 2] Sending to 8,234 users in CST  
[BATCH 3] Sending to 9,876 users in MST
[BATCH 4] Sending to 7,654 users in PST

50,000 users total.

Every single one got a "Good morning!" notification at 3 AM.

I realized the mistake when my own phone buzzed.

3:00 AM. Dark room. "Good morning! 🌅"

I sat up in bed, heart pounding.

The Fallout

Immediate Impact (3:00 AM - 8:00 AM)

Support tickets: 2,847 in 2 hours

App Store reviews:

⭐ "Woke up my baby. Uninstalled." ⭐ "Great app but 3 AM notifications? No." ⭐⭐ "Who does this? Bye."

The Metrics

Uninstall rate: 2% → 18% (900% increase)
App rating: 4.8 → 3.2 (dropped in 12 hours)
Push opt-out rate: 5% → 47%
Active users (next day): -32%

The Emergency Response

1. Stop the Cron Job

# Immediately
crontab -e
# Comment out the line
# 0 8 * * * node send-morning-notifications.js

# Kill any running processes
pkill -f send-morning-notifications

2. Send Apology Notifications

// Apology notification
const sendApology = async (users) => {
  // Only to users who received 3AM notification
  const affectedUsers = await getNotificationRecipients({
    from: '2024-01-15 08:00:00',
    to: '2024-01-15 08:05:00',
    title: 'Good morning!'
  });
  
  for (const user of affectedUsers) {
    await sendPushNotification(user.token, {
      title: "Our sincere apologies 🙏",
      body: "That 3AM notification was a bug. We're sorry! Here's a free week: GOODMORNING",
      data: { screen: "RedeemCode", code: "GOODMORNING" },
      sound: null // SILENT
    });
  }
};

3. Post on Social Media

We messed up. A notification intended for 8 AM went out at 3 AM due to a timezone bug. We're deeply sorry. Here's what happened and how we're fixing it. [Link to incident report]

The Long-Term Fix

1. Timezone-Aware Scheduling

// Use a proper job queue with timezone support
import bull from 'bull';

const morningQueue = new bull('morning-notifications', {
  redis: { host: 'localhost', port: 6379 }
});

// Schedule per timezone
const scheduleForTimezone = async (timezone, hour = 8) => {
  const now = new Date();
  const targetTime = getTargetTimeInUTC(timezone, hour);
  
  if (targetTime > now) {
    await morningQueue.add(
      { timezone, hour },
      { 
        delay: targetTime.getTime() - now.getTime(),
        jobId: `morning_${timezone}`
      }
    );
  }
};

const getTargetTimeInUTC = (timezone, hour) => {
  // Convert 8 AM in timezone to UTC
  const localTime = new Date();
  const formatter = new Intl.DateTimeFormat('en-US', {
    timeZone: timezone,
    hour: 'numeric',
    hour12: false
  });
  
  // Complex logic to calculate UTC time
  // Using libraries like 'moment-timezone' is better
  return moment.tz(timezone).set({ hour, minute: 0, second: 0 }).toDate();
};

2. Rate Limiting Notifications

// Don't send more than X notifications per hour
import rateLimit from 'express-rate-limit';

const notificationLimiter = rateLimit({
  windowMs: 60 * 60 * 1000, // 1 hour
  max: 1, // 1 notification per user per hour
  keyGenerator: (req) => req.body.userId,
  handler: (req, res) => {
    console.log(`Rate limit hit for user ${req.body.userId}`);
    res.status(429).json({ error: 'Too many notifications' });
  }
});

3. Quiet Hours Respect

const shouldSendNotification = (user, now) => {
  const userHour = getLocalHour(user.timezone, now);
  
  // Don't send between 10 PM and 7 AM
  if (userHour >= 22 || userHour < 7) {
    console.log(`Quiet hours: ${userHour}:00, skipping`);
    return false;
  }
  
  // Respect user's notification preferences
  if (user.notificationSettings.morning !== true) {
    return false;
  }
  
  return true;
};

4. Canary Releases for Notifications

// Send to 1% of users first
const canarySend = async (notification, percentage = 1) => {
  const users = await getTargetUsers();
  const canaryUsers = users.slice(0, Math.floor(users.length * (percentage / 100)));
  
  const results = await sendBatch(canaryUsers, notification);
  
  // Monitor for complaints
  const complaintRate = await getComplaintRate(results);
  
  if (complaintRate < 0.01) { // Less than 1% complaints
    // Send to everyone else
    const remainingUsers = users.slice(canaryUsers.length);
    await sendBatch(remainingUsers, notification);
  } else {
    console.error(`High complaint rate: ${complaintRate}%, aborting`);
    await sendApology(canaryUsers);
  }
};

The Recovery

Day 1: Chaos

Uninstall rate peaked at 18%
App store rating dropped to 3.2
Social media angry

Day 2: Apology sent

Uninstall rate dropped to 8%
Rating started recovering (3.5)
Some users accepted apology

Week 1: Feature improvements

Added quiet hours setting
User can choose preferred notification time
"Test notification" button for users

Month 1: Full recovery

Uninstall rate back to 2.5%
Rating recovered to 4.6
Lost users: ~15,000 (never returned)

The Lessons

1. Timezone Math Is Hard

Always use libraries like moment-timezone or date-fns-tz. Never roll your own.

2. Test at All Hours

// Test with different system times
const testTimes = [3, 8, 14, 22]; // 3 AM, 8 AM, 2 PM, 10 PM

for (const hour of testTimes) {
  // Mock system time
  jest.setSystemTime(new Date(2024, 0, 1, hour, 0, 0));
  await sendMorningNotification();
  // Verify behavior
}

3. Rate Limit Everything

Even "good" notifications can be bad at the wrong time.

4. Always Have a Kill Switch

// Feature flag
const morningNotificationsEnabled = await redis.get('feature:morning-notifications');

if (morningNotificationsEnabled !== 'false') {
  await sendMorningNotification();
}

Expo Push Notification Best Practices

// Complete notification service
class NotificationService {
  constructor() {
    this.channels = {
      morning: {
        id: 'morning',
        name: 'Morning reminders',
        importance: 'high',
        sound: 'morning.caf'
      },
      quiet: {
        id: 'quiet',
        name: 'Quiet notifications',
        importance: 'low',
        sound: null
      }
    };
  }
  
  async sendMorningNotification(user) {
    // Check if user exists
    if (!user.expoPushToken) return;
    
    // Check quiet hours
    if (this.isQuietHour(user.timezone)) {
      // Queue for next morning
      await this.queueMorningNotification(user);
      return;
    }
    
    // Check user preferences
    if (!user.settings.morningNotifications) return;
    
    // Rate limit check
    const lastSent = await redis.get(`last_morning:${user.id}`);
    if (lastSent && Date.now() - lastSent < 24 * 60 * 60 * 1000) {
      return; // Already sent today
    }
    
    // Send with priority
    const message = {
      to: user.expoPushToken,
      sound: user.settings.soundEnabled ? 'default' : null,
      title: this.getMorningMessage(user),
      body: this.getMorningBody(user),
      data: { type: 'morning', timestamp: Date.now() },
      priority: 'normal', // Not 'high' to avoid disturbing sleep
      channelId: 'morning'
    };
    
    const result = await Notifications.sendPushNotificationAsync(message);
    
    if (result.status === 'ok') {
      await redis.set(`last_morning:${user.id}`, Date.now(), 'EX', 86400);
    }
    
    return result;
  }
  
  isQuietHour(timezone) {
    const hour = this.getLocalHour(timezone);
    return hour >= 22 || hour < 8; // 10 PM to 8 AM
  }
}

Commands for Testing Notifications

Simulate different timezones

# Run with different TZ
TZ=America/New_York node send-notifications.js
TZ=Europe/London node send-notifications.js
TZ=Asia/Tokyo node send-notifications.js

Monitor notification complaints

SELECT 
  DATE(created_at) as date,
  COUNT(*) as complaints,
  COUNT(DISTINCT user_id) as unique_users
FROM notification_feedback
WHERE type = 'complaint'
GROUP BY DATE(created_at);

Test push delivery

# Expo push tool
expo push:send --tokens tokens.json --message "Test message" --data '{"test":true}'

Conclusion

That 3 AM notification cost us:

15,000 lost users
2 weeks of reputation damage
1 very sleep-deprived engineering team

Now every notification goes through:

Timezone validation
Quiet hours check
Rate limiting
Canary release (for large batches)
User preference check

And most importantly: Never deploy notification systems without testing at all hours.

My phone is now on Do Not Disturb from 10 PM to 8 AM.

And I check cron schedules twice before deploying.