How would you monitor massive changes to AWS infrastructure you just got access to?

Lets say you got called into a situation that is unfolding in real time on your customers AWS Infrastructure. Perhaps they think some creds got leaked and some malicious party is systematically going through and making changes to the system or just as bad perhaps randomly deleting stuff or otherwise causing havoc.

How would you track them? Sounds like the stuff of nightmares but I get calls from CTOs in a cold sweat suffering from stuff like this more often than you would think.

At the most recent Cloud War Games Live Event where we actually drill this stuff in a live sandbox environment one of our participants had an interesting strategy for tracking these changes. Tim, a fairly advanced player, was the first contestant to ask me directly for API access. I was happy to oblige but was curious what he had up his sleeve.

Before the game started and before we started breaking things Tim used a tool known as Google Teraformer to create a version of our entire infrastructure locally. Then, as we proceeded to break things, he simply re-ran terraformer to generate a new copy of the infrastructure which he could diff using some basic text diff tools.

In the real world this makes a lot of sense; The bad guys are not playing by any specific rules. Why should you?

But I am not sure how I feel about this for CWG specifically. I’ll give him points for creativity but that does take a bit of the fun out of things. We will see if other people try the same thing (Though now that I have written about this they probably will).

Wrapping It Up:

I have a few more tricks up my sleeves for monitoring broad changes to the system myself but I will save those for another time.

How would you track changes to your AWS Infrastructure in near real time during an emergency?