“If the only tool you have is a hammer, you tend to see every problem as a nail.”~ Abraham Maslow

Recently I had consulting office hours with a client that had a big Data ETL project. They had several running batch workers that were all erroring out multiple times every night. The processes would automatically restart so the jobs eventually finished but they were paying for a lot of extra compute time because of all the failures and retries.

I dug in and quickly found out why. The original project requirement was that they used AWS Glue / Athena to crawl a bunch of S3 stored data files each night. Reading the files was working great. The problem was when they decided to write the current state to AWS Glue / Athena. After they crawled a certain amount of records they decided to save a “cursor” of sorts that told them which records had already been crawled. This is a great plan... except they decided to save it to AWS Glue.

Now if you don't know about AWS Glue / Athena or working with big data you almost never update it. All the data gets passed in, saved and then read as many times as you need but rarely if ever updated, especially by multiple processes at once. This caused Athena to error out and the whole thing to come crashing down.

After talking to the engineer that wrote it they just thought “Since I am using AWS Glue already for this project to read the data why not use it to track state?” or in other words “I already am using a hammer for this project so every problem must be a nail”.

I tell you all of this not to throw shade at that particular engineer, we all have fallen trap to that line of thinking now and again, but to make you aware of the mental trap you might be falling into if you don’t occasionally put down the hammer, take a step back, and look at what other tools you might have at your disposal.

Also if you are wondering what my solution I came up with I suggested we store the state, which was updated hundreds if not thousands of times a night, in Redis which is a datastore designed for high I/O.

And if you need a bit more help check out my Group Coaching Program or 1 on 1 Consulting at schematical.com ~Cheers