The GitHub Actions Secret That Printed to Logs (And We Rotated All API Keys)

"Secrets are safe in GitHub Actions. They're masked automatically."

That's what I believed.

Until one innocent echo command printed our production AWS secret key in plain text for the world to see.

The Setup

We used GitHub Actions for CI/CD:

# .github/workflows/deploy.yml
name: Deploy to AWS
on: push
env:
  AWS_ACCESS_KEY_ID: $ {{ secrets.AWS_ACCESS_KEY_ID }}
  AWS_SECRET_ACCESS_KEY: $ {{ secrets.AWS_SECRET_ACCESS_KEY }}
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Deploy to S3
        run: |
          aws s3 sync ./build s3://my-bucket

We added a debug step to troubleshoot a failure:

- name: Debug environment
  run: |
    echo "AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID"
    echo "AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY"

The Leak

The run completed. The logs showed:

AWS_ACCESS_KEY_ID: AKIAIOSFODNN7EXAMPLE
AWS_SECRET_ACCESS_KEY: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

Wait — GitHub Actions is supposed to mask secrets automatically.

Why wasn't it masked?

Because GitHub only masks secrets that match the exact value in the secret store.

Our secret value was wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY (with a slash).

When echo printed it, the slash wasn't part of the masking pattern? Actually, GitHub masks based on the exact secret value. It should have worked.

The real issue: Our secret contained special characters, and GitHub's masking is case-sensitive and exact-match only.

But the bigger problem: Anyone with read access to the repo could see the logs.

And our repo was public.

The Aftermath

Within 3 minutes of the workflow run:

CloudTrail showed:

RunInstances - us-east-1 - 50 instances
RunInstances - eu-west-1 - 50 instances
RunInstances - ap-southeast-1 - 50 instances

Someone had scraped the log, found the key, and started mining cryptocurrency.

AWS alerted us via Security Hub.

By the time we revoked the key, they had launched 347 EC2 instances across 8 regions.

Estimated cost for 1 hour: $4,700.

The Emergency Response

1. Revoke the exposed key immediately

aws iam delete-access-key --access-key-id AKIAIOSFODNN7EXAMPLE

2. Terminate unauthorized instances

# List all instances launched after the leak
aws ec2 describe-instances --query "Reservations[].Instances[?LaunchTime>='2024-01-15T10:00:00'].[InstanceId]" --output text | xargs aws ec2 terminate-instances --instance-ids

3. Rotate ALL secrets (not just the exposed one)

# Generate new keys
aws iam create-access-key --user-name deploy-user

# Update GitHub secret
gh secret set AWS_SECRET_ACCESS_KEY --body "$NEW_SECRET"

4. Audit for other leaks

# Search all workflow run logs for patterns
gh run list --limit 100 --json databaseId | jq '.[].databaseId' | xargs -I{} gh run view {} --log | grep -i "secret|key|token"

The Prevention

1. Never echo secrets, even for debugging

# Instead of echo, use ::add-mask:: to manually mask
- name: Debug secret (safe)
  run: |
    echo "::add-mask::$AWS_SECRET_ACCESS_KEY"
    echo "Secret length: $ {#AWS_SECRET_ACCESS_KEY}"

2. Use environment protection rules

# Require approval for production
environment: 
  name: production
  url: https://myapp.com
  required_reviewers:
    - my-username

3. Enable secret scanning in GitHub

GitHub Advanced Security can detect accidentally committed secrets.

4. Use OIDC instead of long-lived keys

# Configure AWS OIDC
- name: Configure AWS credentials
  uses: aws-actions/configure-aws-credentials@v2
  with:
    role-to-assume: arn:aws:iam::123456789:role/github-actions-role
    aws-region: us-east-1
# No long-lived secrets needed!

5. Set up a canary token

# Deploy a fake AWS key that triggers an alert if used
# Using Canarytokens.org

The Cleanup Script

After the incident, we automated key rotation:

#!/bin/bash
# rotate-aws-keys.sh

USER_NAME="github-actions-user"

# Create new key
NEW_KEY=$(aws iam create-access-key --user-name $USER_NAME)

NEW_ID=$(echo $NEW_KEY | jq -r '.AccessKey.AccessKeyId')
NEW_SECRET=$(echo $NEW_KEY | jq -r '.AccessKey.SecretAccessKey')

# Update GitHub secret
echo $NEW_SECRET | gh secret set AWS_SECRET_ACCESS_KEY --repo myorg/myrepo

# Wait for propagation
sleep 60

# Delete old key (find the one not used recently)
OLD_KEY_ID=$(aws iam list-access-keys --user-name $USER_NAME | jq -r '.AccessKeyMetadata[] | select(.Status=="Active") | .AccessKeyId' | head -1)

aws iam delete-access-key --user-name $USER_NAME --access-key-id $OLD_KEY_ID

echo "Rotated key $OLD_KEY_ID -> $NEW_ID"

What I Learned

• Never echo secrets – not even for debugging.
• Use OIDC – no secrets to leak.
• Rotate keys regularly – even if not exposed.
• Monitor CloudTrail aggressively – we caught it early.
• Public repos need extra care – assume hostile readers.

That leaked key cost us $4,700 and a weekend of rotating every single credential in our infrastructure.

Now we have a "no echo of any variable named KEY or SECRET" rule enforced by pre-commit hooks.