A client came to me with a problem “Redis values are being evicted from our Elasticache Cluster but we don’t know which ones. How can we find out which keys are being evicted”. For those of you that don’t know Redis has a built in eviction system so if it starts running out of memory it can start doing garbage collection on older entries. This is great if you designed your system to handle that. It sucks if somehow a valuable key you expect to be there is all of the sudden NOT there.

Their first thought was to check the logs. Great idea except Redis v6 or below does NOT have logs on Elasticache but v6 and above does. We were running below v6 so that was not an option and I am not sure if it logs evictions to the logs.

My solution was to subscribe to the event stream. Redis has the ability to basically do a PUBLISH command that sends out a notification when a change happens on the system. It is called Redis Keyspace Notifications.

First you have to do a CONFIG SET command to tell Redis to start logging. FYI if you are on Elasticache you may not be able to do a config set but don’t worry you can update your instance’s Parameter Group, specifically the notify-keyspace-events setting.

I tested this locally with the following command:

redis-cli config set notify-keyspace-events KEA

A value of KEA will be like drinking from a firehose, it will get you everything. For our purposes we just needed the following 2 events:

x     Expired events (events generated every time a key expires)
e     Evicted events (events generated when a key is evicted for maxmemory)

If you want to listen for others you can grab the ones you want from the Redis Keyspace Notifications Docs.

From there I ran a PSUBSCRIBE command that allowed me to pipe the output to a file with something like this:

redis-cli --csv psubscribe '__key*__:*' > SomeFile.txt 

Yes you have to actively be connected to redis for this to work but it will get the job done, especially for something as simple as just debugging what is being evicted from Redis.

Best of luck!

~Cheers

PS: If you are interested in engaging my services as a Web Application Architect check out my Group Coaching Program or 1 on 1 Consulting at schematical.com and check out my FREE eBook 20 Things You Can DoTo Save Money On Your Amazon Web Services Bill Today