Twas the night before Christmas when all through the facility Not a metric was stirring not even the latency; The servers were humming, all systems were green, The dev team was hoping they'd stay that serene.

And Ma in her kerchief And I in my cap, Had just settled down To a long winter's nap - When from my cell phone There rose such a clatter, I sprang from my bed to see what was the matter

When, what to my wondering eyes should appear, But a notification the servers were down, my greatest fear...

I jumped on my laptop to see what to do My plans of taking Christmas day off had just gone to poo.

As I checked the WAF logs I saw the requests coming in, Is this traffic external or is it coming from within?

The dashboards were all light up in red, I panicked and shouted out all the services that were dead:

"Now, ECS! Now, EC2! Now, RDS and ElastiCache! Oh, APIGateway! Oh, Lambda! Oh, SQS and Batch! Something is a miss, with the firewall! Turned if off and on again, reboot away! reboot away! Reboot away all!"

Slowly but surely metrics turned from dark to bright. I don't quite know what fixed the problem that night. But turning the services off and back on again, Seems to have fixed the problem my friend.

PS: If you want to see me recipte this here is a short video https://www.youtube.com/shorts/bACXv0r5ESU