Ever found yourself fighting a server outage that is costing your company upward of $100,000 an hour? I have.

After watching Hot Seat participants at the Cloud War Game events here is one simple thing I found that I did that I didn’t see the newer players do.

I left myself bread crumbs. So just like Hansle and Grettle I can find my way back to an area I had previously examined. If you ever had been thrown in blind to a server outage where there are literally a thousand or so places where the outage could be placed the outage could be there are likely to be a few red-harrings that throw you off track.

Once you realize you have been going down the wrong path you will want to be able to quickly backtrack. This means following the bread crumbs back to the last place where you had a solid grasp of what was going on.

There are a few ways you can do this. If the servers are on fire and “it” is currently hitting the fan I just open new tabs as I track requests going through the system. I know it's horrible for productivity but it’s nice to have a linear log of what you have looked at.

Once the fire is out I then go ahead and document the bread crumbs and the path I took to find the fire. I do this by transferring them from the tabs to a Google Doc. This gives us solid documentation on the resources affected and the path the issue took to hit the affected resources.

Then other developers and cloud pros can review that doc so they can learn and benefit from that knowledge.

Question For You: What are your “Pro Tips” for aspiring cloud / dev ops professionals?