“Sharding” is not a dirty word

I talk a lot about “Sharding”, and almost every time I bring it up in a meeting with someone that doesn’t know what it is, I almost always get a chuckle.

Why? People often mistake it for a very similar word with a very different meaning.

If you replace the “d” with a “t”, you get a word that means something much different that is normally not discussed in polite company.

But it occurs to me that a lot of people in my audience might not know what “Sharding” means.

What is “Sharding”?

This is the practice of storing different subsets of data on different physical hardware.

Over-Simplified Example:

Imagine you had a database that was overflowing, so you spun up another 2 database servers. Then you stored the records for customers whose names start with “A-K” on server 1, “J-O” on server 2, and “P-Z” on server 3.

This works great when you always have the customer’s name to look up records because you know which server to query. If for any reason you don’t have that info or, if a query spans multiple customers, it gets a bit more complicated.

I did an entire video on how to scale Redis horizontally to save money using sharding on AWS ElastiCache a few years ago, if you want to check it out.

Another thing to note is that “sharding” is often used as a synonym for "partitioning," and while the two are related, depending on the technology implementing the technique, the 2 words have significantly different meanings.

For example, AWS Aurora MySQL allows you to partition data, which instead of having the data stored on separate dedicated databases it keeps all the data on the same database instances but instead has dedicated disk space for the records based in your “partition key”.

In the earlier example, the partition key would be the customer name, but again, that was an oversimplification, so don’t use the customer name as your partition key.

With all that said, if you need help with scaling up your server infrastructure, feel free to reach out and set up a time to chat.